Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data

In [1]:
# Load pickled data
import pickle

training_file = './train.p'
testing_file = './test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

X_coords, X_sizes = train['coords'], train['sizes']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below.

In [2]:
import csv

# Number of training examples
n_train = len(train['labels'])

# Number of testing examples.
n_test = len(test['labels'])

# What's the shape of an traffic sign image?
image_shape = X_train[0].shape

# How many unique classes/labels there are in the dataset.
labelmap = {}
with open('./signnames.csv', 'r') as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        labelmap[int(row['ClassId'])] = row['SignName']
n_classes = len(labelmap)
        
print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.

In [3]:
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline

import random

index = random.randint(0, n_train)
image = X_train[index]

plt.figure(figsize=(1,1))
plt.imshow(image)
print("Description: " + labelmap[y_train[index]])
print("Original image size:")
print(X_sizes[index])
print("Original image bounding box (x1, y1, x2, y2):")
print(X_coords[index])
Description: Dangerous curve to the right
Original image size:
[81 71]
Original image bounding box (x1, y1, x2, y2):
[ 8  7 74 65]

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [4]:
# Pre-processing of the data

import numpy as np
import cv2

# First, let's see how many items of each category we have:
class_index = []
# let's initialize class_index
for index in range(len(y_train)):
    class_index.append([])

for index in range(len(y_train)):
    item_class = y_train[index]
    class_index[item_class].append(index)

# show a histogram for human readibility
plt.hist(y_train, bins='auto')
plt.title("Initial count of images (y) per class (x)")
plt.show()

def modify_image(image):
    """
    Expects the parameter to be a 3-dimensional array with the RGB information
    for the image; will create a new image by modifying the original with random
    modifications, particularly: tilting.
    
    http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html
    
    """
    rows,cols = len(image), len(image[0])
    # tilt [-20,20]
    tilt_angle = -20 + int(random.random() * 40)
    M = cv2.getRotationMatrix2D((cols/2,rows/2),tilt_angle,1)
    out = cv2.warpAffine(image,M,(cols,rows))
    # scale [-2,2] on each point in x, [-2,2] on each point in y
    scale_x_left = -2 + int(random.random() * 4)
    scale_x_right = -2 + int(random.random() * 4)
    scale_y_top = -2 + int(random.random() * 4)
    scale_y_bottom = -2 + int(random.random() * 4)
    pts1 = np.float32([[0,0],[rows,0],[0,cols],[rows,cols]])
    pts2 = np.float32([[scale_y_top,scale_x_left],
                       [rows+scale_y_bottom,scale_x_left],
                       [scale_y_top,cols+scale_x_right],
                       [rows+scale_y_bottom,cols+scale_x_right]])
    M = cv2.getPerspectiveTransform(pts1,pts2)
    out = cv2.warpPerspective(out,M,(cols,rows))
    return np.asarray(out)



# The disparity in image count is striking, we should attempt to equalize the field at least a little bit
# so the difference in example count between different classes isn't as wide
# We will ensure that each class has at least N items, so we'll iterate over the class index and ensure that
# we add the required amount of images; these new images will be created by applying modifications to
# the initial set of images
# Arbitrarily I've chosen 2000 as the minimum number of images given that some classes have as little as 250
# items while others have as much as 2500. Lowering this disparity is what I intend
min_item_count = 2000
extension_to_X_train = []
extension_to_y_train = []
extension_to_X_coords = []
extension_to_X_sizes = []
for current_class in range(n_classes):
    item_index = class_index[current_class]
    item_count = len(item_index)
    if item_count < min_item_count:
        #print("Class: {0}".format(current_class))
        #print(item_count)
        #print(item_index)
        # let's add the new data
        N = min_item_count - item_count
        for i in range(N):
            #print("Index {0}".format(i))
            image_selector = item_index[i % item_count]
            #print("Image selector {0}".format(image_selector))
            selected_image = X_train[image_selector]
            #print(selected_image)
            #plt.figure(figsize=(1,1))
            #plt.imshow(selected_image)
            extension_to_X_train.append(modify_image(selected_image))
            extension_to_y_train.append(y_train[image_selector])
            extension_to_X_coords.append(X_coords[image_selector])
            extension_to_X_sizes.append(X_sizes[image_selector])

# Append extensions
if (len(extension_to_X_train) > 0):
    X_train = np.append(X_train, extension_to_X_train, axis=0)
    y_train = np.append(y_train, extension_to_y_train, axis=0)
    X_coords = np.append(X_coords, extension_to_X_coords, axis=0)
    X_sizes = np.append(X_sizes, extension_to_X_sizes, axis=0)
    n_train = len(X_train)
In [5]:
# Give a count of the modified training data
# show a histogram for human readibility
print("Total item count =", n_train)
plt.hist(y_train, bins='auto')
plt.title("Count of images (y) per class (x) after augmentation")
plt.show()

if (len(extension_to_X_train) > 0):
  index = random.randint(0, len(extension_to_X_train))
  image = extension_to_X_train[index]
  plt.figure(figsize=(1,1))
  plt.imshow(image)
  print("Description: " + labelmap[extension_to_y_train[index]] + " - ", index)
Total item count = 86810
Description: Dangerous curve to the right -  15395

Question 1

Describe how you preprocessed the data. Why did you choose that technique?

Answer:

When I did the histogram to visually count how many items we had per class (label) it was clear that the disparity was very large. The suggestions given in the Step 2 header were a hint towards this being an issue. Initially when I first did the LeNet pass on the original data I realized my validation scores and test scores were low, and a bit random. I decided to try and augment the examples to be a more even count per item class, and settled on having at least 2000 examples on each class.

The way I did this was by making an index of all the images per class. I then counted how many images were there in a give class and if the count was less than 2000 I'd proceed with augmentation. These are the augmentation steps:

  1. Obtain the list of images for this class
  2. Iterate 2000-count(images for this class) times
  3. Take the next image from the list of images for this class (so we use them all to augment the data set)
  4. Modify the image
  5. Add it to the extension list
  6. Add metadata to the extension metadata lists
  7. After done iterating, merge the extension lists with the original data

I then produced a new histogram of the resulting set of images. Since I augmented all the metadata sets as well, the length and order of the data is preserved and would allow for the rest of this exercise to proceed regardless of the augmentation step.

As for the image modification, I decided to make random tweaks along two different operations:

  1. Tilt the image anywhere from -20 degrees to 20 degrees
  2. Change the shape of the image anywhere from -2 pixels to 2 pixels on each corner, on each dimension

I based the transformations on the examples given at http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html

The idea behind this was to ensure that the extended data was different (even if just slightly) to the original data so as to actually help the model see new things.

Note

I am explicitly NOT normalizing the data in this step as I discovered that I had issues with doing so given what I wanted to do next: create 3-channel grayscale representations of the same images, effectively doubling the data set.

In the following code blocks I generate the grayscale set and after that is done I proceed to normalize the data as told in the online class. We want the data to be centered around zero, and hopefully with an normal distribution around zero.

Then I proceed to split the augmented data into a training set and a validation set.

In [6]:
import cv2

#X_train_grayscale = []
#for index in range(n_train):
#    grayscale = cv2.cvtColor(X_train_scaled[index], cv2.COLOR_BGR2GRAY)
#    X_train_grayscale.append(grayscale)
X_train_grayscale = []
for index in range(n_train):
    grayscale = np.zeros_like(X_train[index], dtype=np.float32)
    rgb = X_train[index]
    y = 0.2989 * rgb[:,:,2] + 0.5870 * rgb[:,:,1] + 0.1140 * rgb[:,:,0]
    grayscale[:,:,0] = grayscale[:,:,1] = grayscale[:,:,2] = y.squeeze()
    X_train_grayscale.append(grayscale)


print("Done grayscaling, rendering an example output vs the original:")

index = random.randint(0, n_train)
print("Index: ", index)
image = X_train_grayscale[index]

plt.figure(figsize=(1,1))
plt.imshow(image)
plt.figure(figsize=(1,1))
plt.imshow(X_train[index])
print("Description: " + labelmap[y_train[index]])

# we now have a second set of images which is the same as the cropped-resized images we pre-processed before
# but in a human-balanced grayscale, still 3 channels.
Done grayscaling, rendering an example output vs the original:
Index:  18713
Description: Priority road
In [7]:
# Normalize the data first (centered at 0,0, with a span of 1)
# We know the RGB data has values 0 to 255

def normalize_image(image):
    return (image - 128) / 128

# Normalize the data ensuring the values are float32
X_train_grayscale_normalized = np.array([normalize_image(i) for i in X_train_grayscale])
X_train_normalized = np.array([normalize_image(i) for i in X_train.astype(np.float32)])
X_test_normalized = np.array([normalize_image(i) for i in X_test.astype(np.float32)])
In [8]:
#index = random.randint(0, n_train)
print("Index: ", index)
plt.figure(figsize=(1,1))
plt.imshow(X_train_normalized[index])
plt.figure(figsize=(1,1))
plt.imshow(X_train_grayscale_normalized[index])
print("Example images from normalized data")
Index:  18713
Example images from normalized data
In [9]:
# Let's say we want 80% of the data to be testing data, and 20% to be validation data
# We also want to shuffle the items in the testing set and the validation set
from sklearn.utils import shuffle

train_split_prob = 0.8
X_train_split = []
y_train_split = []
X_validation_split = []
y_validation_split = []
def split_and_append(Xinput, yinput, split_prob, X1, y1, X2, y2):
    """
    Takes the input X,y and appends them to X1,y1 or X2,y2 depending
    on the split_prob value.
    """
    for index in range(len(Xinput)):
        image = Xinput[index]
        classification = yinput[index]
        is_second_split = random.random() >= split_prob
        if is_second_split:
            X2.append(image)
            y2.append(classification)
        else:
            X1.append(image)
            y1.append(classification)

split_and_append(X_train_normalized,
                 y_train,
                 train_split_prob,
                 X_train_split,
                 y_train_split,
                 X_validation_split,
                 y_validation_split)
split_and_append(X_train_grayscale_normalized,
                 y_train,
                 train_split_prob,
                 X_train_split,
                 y_train_split,
                 X_validation_split,
                 y_validation_split)

X_train_split, y_train_split = shuffle(X_train_split, y_train_split)
X_validation_split, y_validation_split = shuffle(X_validation_split, y_validation_split)


print("Done splitting and shuffling items")
expected_len_X = len(X_train_normalized) + len(X_train_grayscale_normalized)
expected_len_y = len(y_train) * 2
len_train_X = len(X_train_split)
len_validation_X = len(X_validation_split)
len_train_y = len(y_train_split)
len_valudation_y = len(y_validation_split)
assert expected_len_X == len_train_X + len_validation_X
assert expected_len_y == len_train_y + len_valudation_y
assert len_train_X == len_train_y
assert len_validation_X == len_valudation_y
print(len_train_X)
print(len_validation_X)

index = random.randint(0, len(X_train_split))
image = X_train_split[index]

plt.figure(figsize=(1,1))
plt.imshow(image)
print("Description: " + labelmap[y_train_split[index]])
Done splitting and shuffling items
138829
34791
Description: Bumpy road

Question 2

Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?

Splitting the training set

We already had one training set and one test set to begin with. I'm leaving the test set as is and intend to use it only once after I'm fully confident of the accuracy of my model based on the test data.

The training set, however, is going to be split in 2 parts:

  • 80% of the data into the actual training set
  • 20% of the data into a new validation set

This partition is achieved by iterating on the input data set (X,y) and splitting it into the 2 buckets (training(X,y) and validation(X,y)). Each data pair (Xi,yi) is assigned to a set by obtaining a random number between 0.0 and 1.0 and choosing one bucket if the random number is below the threshold, or the other bucket if it's above the threshold. This makes it easy to think of partitioning the data as percentages of the input. Note that here we assume that the random number generator gives a relatively uniform distribution of results over the [0.0, 1.0] continuum.

After the training set and validation set have been created, we proceed to shuffle their contents and perform various assertions.

Newly generated data

I decided to take this step to generate grayscale images corresponding to the original+augmented images. We really have 2 data sets at this point:

  • The original color images
  • The grayscale images created from the color images

I'm thinking perhaps training the data on both the color and the grayscale images might help me improve the accuracy of the predictions going forward. I'll try to ensure that the network can decide "this is a yield sign" based on the unique characteristics found in the color image, or the more general strokes seen in the grayscale image.

As for the grayscale, I decided to compute it myself with a human-eye based formula I found online:

y = 0.2989 * red + 0.5870 * green + 0.1140 * blue

There was no big reason for this, I just liked the way it looked better than the built-in algorithms I had at hand.

Ideas that didn't work

I read that the picked data contained cropping information, encoded as the 4 corners of a square in the original image size map. I transformed the cropping information to the coordinate system used by the scaled image (32 x 32) used a PIL image to first crop the area outside the pixels of interest. Since our algorithm assumes a 32 x 32 image, I scaled the image back to the specified dimensions.

I chose this in order to reduce the signal-to-noise ratio of the data we're sending to the network. I'm assuming that the pixels outside of the area of interest are noise and might lower the overall accuracy of our predictions. For example, if all the training images had green or brown from trees in the background, the network might fail to understand that a sign with a blue-sky in the background is equally valid. Furthermore, if we move this to grayscale, the elimination of the surrounding pixels is just as useful. Imagine that for a specific type of traffic sign, such as "yield", we didn't have enough training data. In that case our network will be very sensitive to all the pixels available in the few images we have. This might mean that our network can only recognize "yield" signs when the signal and the noise levels of the test data are similar to those of the training data. I'm assuming that by reducing the noise in the training data we render the noise in the test data less detrimental overall.

My assumptions might be wrong, so not only will I be training my network, I'll be training my own personal experience and the heuristics I'll use going forward.

Answer:

In [10]:
def LeNet(x, keep_probability):    
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1
    
    # Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
    # new_height = (input_height - filter_height + 2 * P)/S + 1
    # 28 = ((32 - fh)/S) + 1
    #(27 * S) + fh= 32
    # S = 1, fh = 5    
    F_W = tf.Variable(tf.truncated_normal((5, 5, 3, 6), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(6))
    strides = [1, 1, 1, 1]
    padding = 'VALID'
    layer1 = tf.nn.conv2d(x, F_W, strides, padding) + F_b

    # Activation 1.
    layer1 = tf.nn.relu(layer1)
    
    # Dropout 1.
    layer1 = tf.nn.dropout(layer1, keep_probability)

    # Pooling 1. Input = 28x28x6. Output = 14x14x6.
    # new_height = (input_height - filter_height)/S + 1
    # 14 = ((28 - fh)/S) + 1
    # (13 * S) + fh = 28
    # S = 2, fh = 2
    ksize=[1, 2, 2, 1]
    strides=[1, 2, 2, 1]
    padding = 'VALID'
    pooling_layer1 = tf.nn.max_pool(layer1, ksize, strides, padding)
    
    # Layer 2a: Convolutional. Input = 28x28x6. Output = 5x5x16.
    # new_height = (input_height - filter_height + 2 * P)/S + 1
    # 5 = ((28 - fh)/S) + 1
    # (4 * S) + fh = 28
    # S = 2, fh = 20
    F_W = tf.Variable(tf.truncated_normal((20, 20, 6, 16), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(16))
    strides = [1, 2, 2, 1]
    padding = 'VALID'
    layer2a = tf.nn.conv2d(layer1, F_W, strides, padding) + F_b
    
    # Flatten 2a. Input = 5x5x16. Output = 400.
    flatten_layer2a = tf.contrib.layers.flatten(layer2a)

    # Layer 2b: Convolutional. Input = 14x14x6. Output = 10x10x16.
    # new_height = (input_height - filter_height + 2 * P)/S + 1
    # 10 = ((14 - fh)/S) + 1
    # (9 * S) + fh = 14
    # S = 1, fh = 5
    F_W = tf.Variable(tf.truncated_normal((5, 5, 6, 16), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(16))
    strides = [1, 1, 1, 1]
    padding = 'VALID'
    layer2b = tf.nn.conv2d(pooling_layer1, F_W, strides, padding) + F_b
    
    # Activation 2b.
    layer2b = tf.nn.relu(layer2b)
    
    # Dropout 2b.
    layer2b = tf.nn.dropout(layer2b, keep_probability)

    # Pooling 2b. Input = 10x10x16. Output = 5x5x16.
    # new_height = (input_height - filter_height)/S + 1
    # 5 = ((10 - fh)/S) + 1
    # (4 * S) + fh = 10
    # S = 2, fh = 2
    ksize=[1, 2, 2, 1]
    strides=[1, 2, 2, 1]
    padding = 'VALID'
    pooling_layer2b = tf.nn.max_pool(layer2b, ksize, strides, padding)

    # Flatten 2b. Input = 5x5x16. Output = 400.
    flatten_layer2b = tf.contrib.layers.flatten(pooling_layer2b)
    
    # Concat layer 2a and layer 2b
    flat_layer2 = tf.concat_v2([flatten_layer2b, flatten_layer2a], 1)
    
    # Layer 3: Fully Connected. Input = 800. Output = 120.
    F_W = tf.Variable(tf.truncated_normal((800, 120), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(120))
    fully_connected = tf.matmul(flat_layer2, F_W) + F_b
    
    # Activation 3.
    fully_connected = tf.nn.relu(fully_connected)
    
    # Dropout 3.
    fully_connected = tf.nn.dropout(fully_connected, keep_probability)

    # Layer 4: Fully Connected. Input = 120. Output = 84.
    F_W = tf.Variable(tf.truncated_normal((120, 84), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(84))
    fully_connected = tf.matmul(fully_connected, F_W) + F_b
    
    # Activation 4.
    fully_connected = tf.nn.relu(fully_connected)

    # Layer 5: Fully Connected. Input = 84. Output = n_classes.
    F_W = tf.Variable(tf.truncated_normal((84, n_classes), mean = mu, stddev = sigma, dtype=tf.float32))
    F_b = tf.Variable(tf.zeros(n_classes))
    logits = tf.matmul(fully_connected, F_W) + F_b
    
    return logits

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

The code above is based on the published baseline model on this problem that was referenced above in this document.

I originally took my LeNet implementation with dropout operations, and ran it as it was. I worked pretty well on the training and validation but not so well on the test data or the real-world data. I then decided to revisit the Sermanet-Yann paper and learn what they did differently. I noticed that the stage-1 data after sub-sampling was being fed directly to the classifier. I had to look this up online because it wasn't clear to me how you could do that while at the same time taking the stage-2 data into the classifier.

Upon closer inspection of the graphics and the description in the whitepaper, I saw that the data flowing from the stage-1 and stage-2 to the classifier was a convolution, and what appeared to be a simple concatenation so I gave it a try.

In order to do that I split the stage-2 into 2 parts:

  • A. The stage-1 data with convolution and flattening.
  • B. The stage-1 data with convolution, pooling, and flattening.

I then took the 2 flat sets and called tf.concat_v2 on them to producing a single set of features for the classifier. I originally tried calling tf.concat but I never managed to make it work the way I wanted it. The resulting set was an 800-long feature set which I then continued processing as in the previous implementation of LeNet, only changing the input size of the classifier from 400 to 800.

Summary of steps

  • Layer 1: Convolutional. Input = 32x32x3. Output = 28x28x6.
  • Activation 1.
  • Dropout 1.
  • Pooling 1. Input = 28x28x6. Output = 14x14x6.
  • Layer 2a: Convolutional. Input = 28x28x6. Output = 5x5x16.
  • Flatten 2a. Input = 5x5x16. Output = 400.
  • Layer 2b: Convolutional. Input = 14x14x6. Output = 10x10x16.
  • Activation 2b.
  • Dropout 2b.
  • Pooling 2b. Input = 10x10x16. Output = 5x5x16.
  • Flatten 2b. Input = 5x5x16. Output = 400.
  • Concat layer 2a and layer 2b
  • Layer 3: Fully Connected. Input = 800. Output = 120.
  • Activation 3.
  • Dropout 3.
  • Layer 4: Fully Connected. Input = 120. Output = 84.
  • Activation 4.
  • Layer 5: Fully Connected. Input = 84. Output = n_classes.
In [11]:
import tensorflow as tf

# Let's initialize the model
x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))
keep_probability = tf.placeholder(tf.float32)
one_hot_y = tf.one_hot(y, n_classes)

adam_learning_rate = 0.0001
#gradient_descent_learning_rate = 0.1

logits = LeNet(x, keep_probability)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=one_hot_y)
#prediction = tf.nn.softmax(logits)
#cross_entropy = -tf.reduce_sum(y * tf.log(prediction), reduction_indices=1)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(adam_learning_rate)
#optimizer = tf.train.GradientDescentOptimizer(gradient_descent_learning_rate)
training_operation = optimizer.minimize(loss_operation)
In [12]:
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()
In [13]:
EPOCHS = 2000
BATCH_SIZE = 2048


def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_probability: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train_split)
    
    print("Training...")
    print()
    for i in range(EPOCHS):
        X_train_split, y_train_split = shuffle(X_train_split, y_train_split)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train_split[offset:end], y_train_split[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_probability: 1.0})
            
        validation_accuracy = evaluate(X_validation_split, y_validation_split)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print()
        
    saver.save(sess, './lenet')
    print("Model saved")
Training...

EPOCH 1 ...
Validation Accuracy = 0.100

EPOCH 2 ...
Validation Accuracy = 0.240

EPOCH 3 ...
Validation Accuracy = 0.410

EPOCH 4 ...
Validation Accuracy = 0.529

EPOCH 5 ...
Validation Accuracy = 0.610

EPOCH 6 ...
Validation Accuracy = 0.664

EPOCH 7 ...
Validation Accuracy = 0.705

EPOCH 8 ...
Validation Accuracy = 0.730

EPOCH 9 ...
Validation Accuracy = 0.758

EPOCH 10 ...
Validation Accuracy = 0.782

EPOCH 11 ...
Validation Accuracy = 0.800

EPOCH 12 ...
Validation Accuracy = 0.810

EPOCH 13 ...
Validation Accuracy = 0.823

EPOCH 14 ...
Validation Accuracy = 0.831

EPOCH 15 ...
Validation Accuracy = 0.842

EPOCH 16 ...
Validation Accuracy = 0.847

EPOCH 17 ...
Validation Accuracy = 0.853

EPOCH 18 ...
Validation Accuracy = 0.859

EPOCH 19 ...
Validation Accuracy = 0.867

EPOCH 20 ...
Validation Accuracy = 0.869

EPOCH 21 ...
Validation Accuracy = 0.877

EPOCH 22 ...
Validation Accuracy = 0.879

EPOCH 23 ...
Validation Accuracy = 0.883

EPOCH 24 ...
Validation Accuracy = 0.887

EPOCH 25 ...
Validation Accuracy = 0.891

EPOCH 26 ...
Validation Accuracy = 0.891

EPOCH 27 ...
Validation Accuracy = 0.893

EPOCH 28 ...
Validation Accuracy = 0.899

EPOCH 29 ...
Validation Accuracy = 0.902

EPOCH 30 ...
Validation Accuracy = 0.905

EPOCH 31 ...
Validation Accuracy = 0.907

EPOCH 32 ...
Validation Accuracy = 0.909

EPOCH 33 ...
Validation Accuracy = 0.912

EPOCH 34 ...
Validation Accuracy = 0.913

EPOCH 35 ...
Validation Accuracy = 0.917

EPOCH 36 ...
Validation Accuracy = 0.919

EPOCH 37 ...
Validation Accuracy = 0.919

EPOCH 38 ...
Validation Accuracy = 0.923

EPOCH 39 ...
Validation Accuracy = 0.924

EPOCH 40 ...
Validation Accuracy = 0.926

EPOCH 41 ...
Validation Accuracy = 0.927

EPOCH 42 ...
Validation Accuracy = 0.929

EPOCH 43 ...
Validation Accuracy = 0.933

EPOCH 44 ...
Validation Accuracy = 0.931

EPOCH 45 ...
Validation Accuracy = 0.935

EPOCH 46 ...
Validation Accuracy = 0.936

EPOCH 47 ...
Validation Accuracy = 0.938

EPOCH 48 ...
Validation Accuracy = 0.939

EPOCH 49 ...
Validation Accuracy = 0.941

EPOCH 50 ...
Validation Accuracy = 0.942

EPOCH 51 ...
Validation Accuracy = 0.943

EPOCH 52 ...
Validation Accuracy = 0.944

EPOCH 53 ...
Validation Accuracy = 0.945

EPOCH 54 ...
Validation Accuracy = 0.947

EPOCH 55 ...
Validation Accuracy = 0.946

EPOCH 56 ...
Validation Accuracy = 0.948

EPOCH 57 ...
Validation Accuracy = 0.948

EPOCH 58 ...
Validation Accuracy = 0.949

EPOCH 59 ...
Validation Accuracy = 0.952

EPOCH 60 ...
Validation Accuracy = 0.951

EPOCH 61 ...
Validation Accuracy = 0.952

EPOCH 62 ...
Validation Accuracy = 0.953

EPOCH 63 ...
Validation Accuracy = 0.955

EPOCH 64 ...
Validation Accuracy = 0.953

EPOCH 65 ...
Validation Accuracy = 0.956

EPOCH 66 ...
Validation Accuracy = 0.957

EPOCH 67 ...
Validation Accuracy = 0.957

EPOCH 68 ...
Validation Accuracy = 0.958

EPOCH 69 ...
Validation Accuracy = 0.959

EPOCH 70 ...
Validation Accuracy = 0.959

EPOCH 71 ...
Validation Accuracy = 0.961

EPOCH 72 ...
Validation Accuracy = 0.961

EPOCH 73 ...
Validation Accuracy = 0.960

EPOCH 74 ...
Validation Accuracy = 0.962

EPOCH 75 ...
Validation Accuracy = 0.963

EPOCH 76 ...
Validation Accuracy = 0.963

EPOCH 77 ...
Validation Accuracy = 0.964

EPOCH 78 ...
Validation Accuracy = 0.965

EPOCH 79 ...
Validation Accuracy = 0.964

EPOCH 80 ...
Validation Accuracy = 0.966

EPOCH 81 ...
Validation Accuracy = 0.965

EPOCH 82 ...
Validation Accuracy = 0.966

EPOCH 83 ...
Validation Accuracy = 0.967

EPOCH 84 ...
Validation Accuracy = 0.967

EPOCH 85 ...
Validation Accuracy = 0.968

EPOCH 86 ...
Validation Accuracy = 0.968

EPOCH 87 ...
Validation Accuracy = 0.969

EPOCH 88 ...
Validation Accuracy = 0.969

EPOCH 89 ...
Validation Accuracy = 0.970

EPOCH 90 ...
Validation Accuracy = 0.969

EPOCH 91 ...
Validation Accuracy = 0.971

EPOCH 92 ...
Validation Accuracy = 0.970

EPOCH 93 ...
Validation Accuracy = 0.970

EPOCH 94 ...
Validation Accuracy = 0.972

EPOCH 95 ...
Validation Accuracy = 0.973

EPOCH 96 ...
Validation Accuracy = 0.972

EPOCH 97 ...
Validation Accuracy = 0.973

EPOCH 98 ...
Validation Accuracy = 0.972

EPOCH 99 ...
Validation Accuracy = 0.972

EPOCH 100 ...
Validation Accuracy = 0.974

EPOCH 101 ...
Validation Accuracy = 0.972

EPOCH 102 ...
Validation Accuracy = 0.975

EPOCH 103 ...
Validation Accuracy = 0.975

EPOCH 104 ...
Validation Accuracy = 0.975

EPOCH 105 ...
Validation Accuracy = 0.974

EPOCH 106 ...
Validation Accuracy = 0.975

EPOCH 107 ...
Validation Accuracy = 0.976

EPOCH 108 ...
Validation Accuracy = 0.976

EPOCH 109 ...
Validation Accuracy = 0.975

EPOCH 110 ...
Validation Accuracy = 0.976

EPOCH 111 ...
Validation Accuracy = 0.976

EPOCH 112 ...
Validation Accuracy = 0.977

EPOCH 113 ...
Validation Accuracy = 0.977

EPOCH 114 ...
Validation Accuracy = 0.977

EPOCH 115 ...
Validation Accuracy = 0.978

EPOCH 116 ...
Validation Accuracy = 0.978

EPOCH 117 ...
Validation Accuracy = 0.977

EPOCH 118 ...
Validation Accuracy = 0.978

EPOCH 119 ...
Validation Accuracy = 0.978

EPOCH 120 ...
Validation Accuracy = 0.977

EPOCH 121 ...
Validation Accuracy = 0.979

EPOCH 122 ...
Validation Accuracy = 0.980

EPOCH 123 ...
Validation Accuracy = 0.980

EPOCH 124 ...
Validation Accuracy = 0.980

EPOCH 125 ...
Validation Accuracy = 0.979

EPOCH 126 ...
Validation Accuracy = 0.980

EPOCH 127 ...
Validation Accuracy = 0.980

EPOCH 128 ...
Validation Accuracy = 0.980

EPOCH 129 ...
Validation Accuracy = 0.980

EPOCH 130 ...
Validation Accuracy = 0.981

EPOCH 131 ...
Validation Accuracy = 0.981

EPOCH 132 ...
Validation Accuracy = 0.981

EPOCH 133 ...
Validation Accuracy = 0.982

EPOCH 134 ...
Validation Accuracy = 0.981

EPOCH 135 ...
Validation Accuracy = 0.981

EPOCH 136 ...
Validation Accuracy = 0.982

EPOCH 137 ...
Validation Accuracy = 0.982

EPOCH 138 ...
Validation Accuracy = 0.982

EPOCH 139 ...
Validation Accuracy = 0.982

EPOCH 140 ...
Validation Accuracy = 0.982

EPOCH 141 ...
Validation Accuracy = 0.982

EPOCH 142 ...
Validation Accuracy = 0.982

EPOCH 143 ...
Validation Accuracy = 0.982

EPOCH 144 ...
Validation Accuracy = 0.984

EPOCH 145 ...
Validation Accuracy = 0.983

EPOCH 146 ...
Validation Accuracy = 0.983

EPOCH 147 ...
Validation Accuracy = 0.984

EPOCH 148 ...
Validation Accuracy = 0.984

EPOCH 149 ...
Validation Accuracy = 0.984

EPOCH 150 ...
Validation Accuracy = 0.983

EPOCH 151 ...
Validation Accuracy = 0.983

EPOCH 152 ...
Validation Accuracy = 0.984

EPOCH 153 ...
Validation Accuracy = 0.984

EPOCH 154 ...
Validation Accuracy = 0.985

EPOCH 155 ...
Validation Accuracy = 0.985

EPOCH 156 ...
Validation Accuracy = 0.985

EPOCH 157 ...
Validation Accuracy = 0.985

EPOCH 158 ...
Validation Accuracy = 0.984

EPOCH 159 ...
Validation Accuracy = 0.985

EPOCH 160 ...
Validation Accuracy = 0.984

EPOCH 161 ...
Validation Accuracy = 0.985

EPOCH 162 ...
Validation Accuracy = 0.986

EPOCH 163 ...
Validation Accuracy = 0.985

EPOCH 164 ...
Validation Accuracy = 0.986

EPOCH 165 ...
Validation Accuracy = 0.985

EPOCH 166 ...
Validation Accuracy = 0.985

EPOCH 167 ...
Validation Accuracy = 0.985

EPOCH 168 ...
Validation Accuracy = 0.986

EPOCH 169 ...
Validation Accuracy = 0.986

EPOCH 170 ...
Validation Accuracy = 0.986

EPOCH 171 ...
Validation Accuracy = 0.986

EPOCH 172 ...
Validation Accuracy = 0.986

EPOCH 173 ...
Validation Accuracy = 0.986

EPOCH 174 ...
Validation Accuracy = 0.986

EPOCH 175 ...
Validation Accuracy = 0.986

EPOCH 176 ...
Validation Accuracy = 0.987

EPOCH 177 ...
Validation Accuracy = 0.986

EPOCH 178 ...
Validation Accuracy = 0.987

EPOCH 179 ...
Validation Accuracy = 0.987

EPOCH 180 ...
Validation Accuracy = 0.986

EPOCH 181 ...
Validation Accuracy = 0.987

EPOCH 182 ...
Validation Accuracy = 0.987

EPOCH 183 ...
Validation Accuracy = 0.987

EPOCH 184 ...
Validation Accuracy = 0.987

EPOCH 185 ...
Validation Accuracy = 0.987

EPOCH 186 ...
Validation Accuracy = 0.987

EPOCH 187 ...
Validation Accuracy = 0.987

EPOCH 188 ...
Validation Accuracy = 0.987

EPOCH 189 ...
Validation Accuracy = 0.987

EPOCH 190 ...
Validation Accuracy = 0.988

EPOCH 191 ...
Validation Accuracy = 0.988

EPOCH 192 ...
Validation Accuracy = 0.987

EPOCH 193 ...
Validation Accuracy = 0.988

EPOCH 194 ...
Validation Accuracy = 0.987

EPOCH 195 ...
Validation Accuracy = 0.987

EPOCH 196 ...
Validation Accuracy = 0.988

EPOCH 197 ...
Validation Accuracy = 0.988

EPOCH 198 ...
Validation Accuracy = 0.988

EPOCH 199 ...
Validation Accuracy = 0.988

EPOCH 200 ...
Validation Accuracy = 0.988

EPOCH 201 ...
Validation Accuracy = 0.988

EPOCH 202 ...
Validation Accuracy = 0.989

EPOCH 203 ...
Validation Accuracy = 0.989

EPOCH 204 ...
Validation Accuracy = 0.989

EPOCH 205 ...
Validation Accuracy = 0.989

EPOCH 206 ...
Validation Accuracy = 0.988

EPOCH 207 ...
Validation Accuracy = 0.989

EPOCH 208 ...
Validation Accuracy = 0.989

EPOCH 209 ...
Validation Accuracy = 0.989

EPOCH 210 ...
Validation Accuracy = 0.989

EPOCH 211 ...
Validation Accuracy = 0.989

EPOCH 212 ...
Validation Accuracy = 0.989

EPOCH 213 ...
Validation Accuracy = 0.989

EPOCH 214 ...
Validation Accuracy = 0.989

EPOCH 215 ...
Validation Accuracy = 0.989

EPOCH 216 ...
Validation Accuracy = 0.989

EPOCH 217 ...
Validation Accuracy = 0.989

EPOCH 218 ...
Validation Accuracy = 0.988

EPOCH 219 ...
Validation Accuracy = 0.989

EPOCH 220 ...
Validation Accuracy = 0.989

EPOCH 221 ...
Validation Accuracy = 0.989

EPOCH 222 ...
Validation Accuracy = 0.990

EPOCH 223 ...
Validation Accuracy = 0.990

EPOCH 224 ...
Validation Accuracy = 0.991

EPOCH 225 ...
Validation Accuracy = 0.990

EPOCH 226 ...
Validation Accuracy = 0.990

EPOCH 227 ...
Validation Accuracy = 0.990

EPOCH 228 ...
Validation Accuracy = 0.990

EPOCH 229 ...
Validation Accuracy = 0.990

EPOCH 230 ...
Validation Accuracy = 0.990

EPOCH 231 ...
Validation Accuracy = 0.990

EPOCH 232 ...
Validation Accuracy = 0.990

EPOCH 233 ...
Validation Accuracy = 0.990

EPOCH 234 ...
Validation Accuracy = 0.990

EPOCH 235 ...
Validation Accuracy = 0.990

EPOCH 236 ...
Validation Accuracy = 0.990

EPOCH 237 ...
Validation Accuracy = 0.991

EPOCH 238 ...
Validation Accuracy = 0.990

EPOCH 239 ...
Validation Accuracy = 0.991

EPOCH 240 ...
Validation Accuracy = 0.990

EPOCH 241 ...
Validation Accuracy = 0.990

EPOCH 242 ...
Validation Accuracy = 0.991

EPOCH 243 ...
Validation Accuracy = 0.989

EPOCH 244 ...
Validation Accuracy = 0.990

EPOCH 245 ...
Validation Accuracy = 0.989

EPOCH 246 ...
Validation Accuracy = 0.990

EPOCH 247 ...
Validation Accuracy = 0.991

EPOCH 248 ...
Validation Accuracy = 0.990

EPOCH 249 ...
Validation Accuracy = 0.991

EPOCH 250 ...
Validation Accuracy = 0.991

EPOCH 251 ...
Validation Accuracy = 0.991

EPOCH 252 ...
Validation Accuracy = 0.990

EPOCH 253 ...
Validation Accuracy = 0.991

EPOCH 254 ...
Validation Accuracy = 0.991

EPOCH 255 ...
Validation Accuracy = 0.991

EPOCH 256 ...
Validation Accuracy = 0.991

EPOCH 257 ...
Validation Accuracy = 0.990

EPOCH 258 ...
Validation Accuracy = 0.991

EPOCH 259 ...
Validation Accuracy = 0.991

EPOCH 260 ...
Validation Accuracy = 0.991

EPOCH 261 ...
Validation Accuracy = 0.991

EPOCH 262 ...
Validation Accuracy = 0.991

EPOCH 263 ...
Validation Accuracy = 0.991

EPOCH 264 ...
Validation Accuracy = 0.991

EPOCH 265 ...
Validation Accuracy = 0.991

EPOCH 266 ...
Validation Accuracy = 0.985

EPOCH 267 ...
Validation Accuracy = 0.990

EPOCH 268 ...
Validation Accuracy = 0.991

EPOCH 269 ...
Validation Accuracy = 0.992

EPOCH 270 ...
Validation Accuracy = 0.992

EPOCH 271 ...
Validation Accuracy = 0.991

EPOCH 272 ...
Validation Accuracy = 0.991

EPOCH 273 ...
Validation Accuracy = 0.992

EPOCH 274 ...
Validation Accuracy = 0.991

EPOCH 275 ...
Validation Accuracy = 0.992

EPOCH 276 ...
Validation Accuracy = 0.992

EPOCH 277 ...
Validation Accuracy = 0.991

EPOCH 278 ...
Validation Accuracy = 0.992

EPOCH 279 ...
Validation Accuracy = 0.991

EPOCH 280 ...
Validation Accuracy = 0.991

EPOCH 281 ...
Validation Accuracy = 0.992

EPOCH 282 ...
Validation Accuracy = 0.992

EPOCH 283 ...
Validation Accuracy = 0.992

EPOCH 284 ...
Validation Accuracy = 0.992

EPOCH 285 ...
Validation Accuracy = 0.992

EPOCH 286 ...
Validation Accuracy = 0.992

EPOCH 287 ...
Validation Accuracy = 0.991

EPOCH 288 ...
Validation Accuracy = 0.992

EPOCH 289 ...
Validation Accuracy = 0.992

EPOCH 290 ...
Validation Accuracy = 0.991

EPOCH 291 ...
Validation Accuracy = 0.991

EPOCH 292 ...
Validation Accuracy = 0.992

EPOCH 293 ...
Validation Accuracy = 0.985

EPOCH 294 ...
Validation Accuracy = 0.989

EPOCH 295 ...
Validation Accuracy = 0.990

EPOCH 296 ...
Validation Accuracy = 0.992

EPOCH 297 ...
Validation Accuracy = 0.992

EPOCH 298 ...
Validation Accuracy = 0.992

EPOCH 299 ...
Validation Accuracy = 0.992

EPOCH 300 ...
Validation Accuracy = 0.992

EPOCH 301 ...
Validation Accuracy = 0.992

EPOCH 302 ...
Validation Accuracy = 0.992

EPOCH 303 ...
Validation Accuracy = 0.992

EPOCH 304 ...
Validation Accuracy = 0.992

EPOCH 305 ...
Validation Accuracy = 0.992

EPOCH 306 ...
Validation Accuracy = 0.992

EPOCH 307 ...
Validation Accuracy = 0.992

EPOCH 308 ...
Validation Accuracy = 0.992

EPOCH 309 ...
Validation Accuracy = 0.992

EPOCH 310 ...
Validation Accuracy = 0.992

EPOCH 311 ...
Validation Accuracy = 0.992

EPOCH 312 ...
Validation Accuracy = 0.992

EPOCH 313 ...
Validation Accuracy = 0.992

EPOCH 314 ...
Validation Accuracy = 0.992

EPOCH 315 ...
Validation Accuracy = 0.992

EPOCH 316 ...
Validation Accuracy = 0.992

EPOCH 317 ...
Validation Accuracy = 0.992

EPOCH 318 ...
Validation Accuracy = 0.992

EPOCH 319 ...
Validation Accuracy = 0.993

EPOCH 320 ...
Validation Accuracy = 0.992

EPOCH 321 ...
Validation Accuracy = 0.992

EPOCH 322 ...
Validation Accuracy = 0.992

EPOCH 323 ...
Validation Accuracy = 0.992

EPOCH 324 ...
Validation Accuracy = 0.992

EPOCH 325 ...
Validation Accuracy = 0.992

EPOCH 326 ...
Validation Accuracy = 0.992

EPOCH 327 ...
Validation Accuracy = 0.992

EPOCH 328 ...
Validation Accuracy = 0.992

EPOCH 329 ...
Validation Accuracy = 0.992

EPOCH 330 ...
Validation Accuracy = 0.990

EPOCH 331 ...
Validation Accuracy = 0.988

EPOCH 332 ...
Validation Accuracy = 0.990

EPOCH 333 ...
Validation Accuracy = 0.993

EPOCH 334 ...
Validation Accuracy = 0.993

EPOCH 335 ...
Validation Accuracy = 0.993

EPOCH 336 ...
Validation Accuracy = 0.993

EPOCH 337 ...
Validation Accuracy = 0.993

EPOCH 338 ...
Validation Accuracy = 0.993

EPOCH 339 ...
Validation Accuracy = 0.993

EPOCH 340 ...
Validation Accuracy = 0.993

EPOCH 341 ...
Validation Accuracy = 0.993

EPOCH 342 ...
Validation Accuracy = 0.993

EPOCH 343 ...
Validation Accuracy = 0.993

EPOCH 344 ...
Validation Accuracy = 0.993

EPOCH 345 ...
Validation Accuracy = 0.993

EPOCH 346 ...
Validation Accuracy = 0.993

EPOCH 347 ...
Validation Accuracy = 0.993

EPOCH 348 ...
Validation Accuracy = 0.993

EPOCH 349 ...
Validation Accuracy = 0.993

EPOCH 350 ...
Validation Accuracy = 0.993

EPOCH 351 ...
Validation Accuracy = 0.993

EPOCH 352 ...
Validation Accuracy = 0.993

EPOCH 353 ...
Validation Accuracy = 0.993

EPOCH 354 ...
Validation Accuracy = 0.993

EPOCH 355 ...
Validation Accuracy = 0.993

EPOCH 356 ...
Validation Accuracy = 0.993

EPOCH 357 ...
Validation Accuracy = 0.993

EPOCH 358 ...
Validation Accuracy = 0.993

EPOCH 359 ...
Validation Accuracy = 0.993

EPOCH 360 ...
Validation Accuracy = 0.993

EPOCH 361 ...
Validation Accuracy = 0.993

EPOCH 362 ...
Validation Accuracy = 0.993

EPOCH 363 ...
Validation Accuracy = 0.993

EPOCH 364 ...
Validation Accuracy = 0.993

EPOCH 365 ...
Validation Accuracy = 0.987

EPOCH 366 ...
Validation Accuracy = 0.986

EPOCH 367 ...
Validation Accuracy = 0.991

EPOCH 368 ...
Validation Accuracy = 0.993

EPOCH 369 ...
Validation Accuracy = 0.993

EPOCH 370 ...
Validation Accuracy = 0.993

EPOCH 371 ...
Validation Accuracy = 0.993

EPOCH 372 ...
Validation Accuracy = 0.993

EPOCH 373 ...
Validation Accuracy = 0.993

EPOCH 374 ...
Validation Accuracy = 0.993

EPOCH 375 ...
Validation Accuracy = 0.993

EPOCH 376 ...
Validation Accuracy = 0.993

EPOCH 377 ...
Validation Accuracy = 0.993

EPOCH 378 ...
Validation Accuracy = 0.993

EPOCH 379 ...
Validation Accuracy = 0.993

EPOCH 380 ...
Validation Accuracy = 0.993

EPOCH 381 ...
Validation Accuracy = 0.993

EPOCH 382 ...
Validation Accuracy = 0.993

EPOCH 383 ...
Validation Accuracy = 0.993

EPOCH 384 ...
Validation Accuracy = 0.993

EPOCH 385 ...
Validation Accuracy = 0.993

EPOCH 386 ...
Validation Accuracy = 0.993

EPOCH 387 ...
Validation Accuracy = 0.993

EPOCH 388 ...
Validation Accuracy = 0.993

EPOCH 389 ...
Validation Accuracy = 0.993

EPOCH 390 ...
Validation Accuracy = 0.993

EPOCH 391 ...
Validation Accuracy = 0.993

EPOCH 392 ...
Validation Accuracy = 0.993

EPOCH 393 ...
Validation Accuracy = 0.993

EPOCH 394 ...
Validation Accuracy = 0.993

EPOCH 395 ...
Validation Accuracy = 0.993

EPOCH 396 ...
Validation Accuracy = 0.993

EPOCH 397 ...
Validation Accuracy = 0.993

EPOCH 398 ...
Validation Accuracy = 0.993

EPOCH 399 ...
Validation Accuracy = 0.993

EPOCH 400 ...
Validation Accuracy = 0.992

EPOCH 401 ...
Validation Accuracy = 0.989

EPOCH 402 ...
Validation Accuracy = 0.988

EPOCH 403 ...
Validation Accuracy = 0.992

EPOCH 404 ...
Validation Accuracy = 0.993

EPOCH 405 ...
Validation Accuracy = 0.993

EPOCH 406 ...
Validation Accuracy = 0.993

EPOCH 407 ...
Validation Accuracy = 0.993

EPOCH 408 ...
Validation Accuracy = 0.993

EPOCH 409 ...
Validation Accuracy = 0.993

EPOCH 410 ...
Validation Accuracy = 0.993

EPOCH 411 ...
Validation Accuracy = 0.993

EPOCH 412 ...
Validation Accuracy = 0.993

EPOCH 413 ...
Validation Accuracy = 0.993

EPOCH 414 ...
Validation Accuracy = 0.993

EPOCH 415 ...
Validation Accuracy = 0.993

EPOCH 416 ...
Validation Accuracy = 0.993

EPOCH 417 ...
Validation Accuracy = 0.993

EPOCH 418 ...
Validation Accuracy = 0.993

EPOCH 419 ...
Validation Accuracy = 0.993

EPOCH 420 ...
Validation Accuracy = 0.993

EPOCH 421 ...
Validation Accuracy = 0.993

EPOCH 422 ...
Validation Accuracy = 0.994

EPOCH 423 ...
Validation Accuracy = 0.993

EPOCH 424 ...
Validation Accuracy = 0.994

EPOCH 425 ...
Validation Accuracy = 0.993

EPOCH 426 ...
Validation Accuracy = 0.993

EPOCH 427 ...
Validation Accuracy = 0.993

EPOCH 428 ...
Validation Accuracy = 0.994

EPOCH 429 ...
Validation Accuracy = 0.993

EPOCH 430 ...
Validation Accuracy = 0.993

EPOCH 431 ...
Validation Accuracy = 0.993

EPOCH 432 ...
Validation Accuracy = 0.993

EPOCH 433 ...
Validation Accuracy = 0.993

EPOCH 434 ...
Validation Accuracy = 0.993

EPOCH 435 ...
Validation Accuracy = 0.993

EPOCH 436 ...
Validation Accuracy = 0.993

EPOCH 437 ...
Validation Accuracy = 0.993

EPOCH 438 ...
Validation Accuracy = 0.993

EPOCH 439 ...
Validation Accuracy = 0.993

EPOCH 440 ...
Validation Accuracy = 0.993

EPOCH 441 ...
Validation Accuracy = 0.991

EPOCH 442 ...
Validation Accuracy = 0.988

EPOCH 443 ...
Validation Accuracy = 0.992

EPOCH 444 ...
Validation Accuracy = 0.993

EPOCH 445 ...
Validation Accuracy = 0.993

EPOCH 446 ...
Validation Accuracy = 0.993

EPOCH 447 ...
Validation Accuracy = 0.994

EPOCH 448 ...
Validation Accuracy = 0.993

EPOCH 449 ...
Validation Accuracy = 0.994

EPOCH 450 ...
Validation Accuracy = 0.994

EPOCH 451 ...
Validation Accuracy = 0.994

EPOCH 452 ...
Validation Accuracy = 0.994

EPOCH 453 ...
Validation Accuracy = 0.994

EPOCH 454 ...
Validation Accuracy = 0.994

EPOCH 455 ...
Validation Accuracy = 0.994

EPOCH 456 ...
Validation Accuracy = 0.994

EPOCH 457 ...
Validation Accuracy = 0.994

EPOCH 458 ...
Validation Accuracy = 0.994

EPOCH 459 ...
Validation Accuracy = 0.994

EPOCH 460 ...
Validation Accuracy = 0.994

EPOCH 461 ...
Validation Accuracy = 0.994

EPOCH 462 ...
Validation Accuracy = 0.994

EPOCH 463 ...
Validation Accuracy = 0.994

EPOCH 464 ...
Validation Accuracy = 0.994

EPOCH 465 ...
Validation Accuracy = 0.993

EPOCH 466 ...
Validation Accuracy = 0.994

EPOCH 467 ...
Validation Accuracy = 0.994

EPOCH 468 ...
Validation Accuracy = 0.994

EPOCH 469 ...
Validation Accuracy = 0.994

EPOCH 470 ...
Validation Accuracy = 0.994

EPOCH 471 ...
Validation Accuracy = 0.993

EPOCH 472 ...
Validation Accuracy = 0.994

EPOCH 473 ...
Validation Accuracy = 0.993

EPOCH 474 ...
Validation Accuracy = 0.993

EPOCH 475 ...
Validation Accuracy = 0.993

EPOCH 476 ...
Validation Accuracy = 0.993

EPOCH 477 ...
Validation Accuracy = 0.993

EPOCH 478 ...
Validation Accuracy = 0.993

EPOCH 479 ...
Validation Accuracy = 0.994

EPOCH 480 ...
Validation Accuracy = 0.993

EPOCH 481 ...
Validation Accuracy = 0.994

EPOCH 482 ...
Validation Accuracy = 0.993

EPOCH 483 ...
Validation Accuracy = 0.994

EPOCH 484 ...
Validation Accuracy = 0.994

EPOCH 485 ...
Validation Accuracy = 0.993

EPOCH 486 ...
Validation Accuracy = 0.993

EPOCH 487 ...
Validation Accuracy = 0.994

EPOCH 488 ...
Validation Accuracy = 0.993

EPOCH 489 ...
Validation Accuracy = 0.990

EPOCH 490 ...
Validation Accuracy = 0.990

EPOCH 491 ...
Validation Accuracy = 0.991

EPOCH 492 ...
Validation Accuracy = 0.993

EPOCH 493 ...
Validation Accuracy = 0.993

EPOCH 494 ...
Validation Accuracy = 0.993

EPOCH 495 ...
Validation Accuracy = 0.994

EPOCH 496 ...
Validation Accuracy = 0.993

EPOCH 497 ...
Validation Accuracy = 0.994

EPOCH 498 ...
Validation Accuracy = 0.994

EPOCH 499 ...
Validation Accuracy = 0.993

EPOCH 500 ...
Validation Accuracy = 0.993

EPOCH 501 ...
Validation Accuracy = 0.993

EPOCH 502 ...
Validation Accuracy = 0.993

EPOCH 503 ...
Validation Accuracy = 0.993

EPOCH 504 ...
Validation Accuracy = 0.994

EPOCH 505 ...
Validation Accuracy = 0.994

EPOCH 506 ...
Validation Accuracy = 0.994

EPOCH 507 ...
Validation Accuracy = 0.994

EPOCH 508 ...
Validation Accuracy = 0.993

EPOCH 509 ...
Validation Accuracy = 0.994

EPOCH 510 ...
Validation Accuracy = 0.994

EPOCH 511 ...
Validation Accuracy = 0.994

EPOCH 512 ...
Validation Accuracy = 0.994

EPOCH 513 ...
Validation Accuracy = 0.994

EPOCH 514 ...
Validation Accuracy = 0.994

EPOCH 515 ...
Validation Accuracy = 0.994

EPOCH 516 ...
Validation Accuracy = 0.994

EPOCH 517 ...
Validation Accuracy = 0.994

EPOCH 518 ...
Validation Accuracy = 0.993

EPOCH 519 ...
Validation Accuracy = 0.994

EPOCH 520 ...
Validation Accuracy = 0.994

EPOCH 521 ...
Validation Accuracy = 0.994

EPOCH 522 ...
Validation Accuracy = 0.994

EPOCH 523 ...
Validation Accuracy = 0.994

EPOCH 524 ...
Validation Accuracy = 0.994

EPOCH 525 ...
Validation Accuracy = 0.994

EPOCH 526 ...
Validation Accuracy = 0.994

EPOCH 527 ...
Validation Accuracy = 0.994

EPOCH 528 ...
Validation Accuracy = 0.994

EPOCH 529 ...
Validation Accuracy = 0.993

EPOCH 530 ...
Validation Accuracy = 0.994

EPOCH 531 ...
Validation Accuracy = 0.994

EPOCH 532 ...
Validation Accuracy = 0.994

EPOCH 533 ...
Validation Accuracy = 0.994

EPOCH 534 ...
Validation Accuracy = 0.993

EPOCH 535 ...
Validation Accuracy = 0.994

EPOCH 536 ...
Validation Accuracy = 0.994

EPOCH 537 ...
Validation Accuracy = 0.993

EPOCH 538 ...
Validation Accuracy = 0.994

EPOCH 539 ...
Validation Accuracy = 0.994

EPOCH 540 ...
Validation Accuracy = 0.994

EPOCH 541 ...
Validation Accuracy = 0.988

EPOCH 542 ...
Validation Accuracy = 0.993

EPOCH 543 ...
Validation Accuracy = 0.993

EPOCH 544 ...
Validation Accuracy = 0.994

EPOCH 545 ...
Validation Accuracy = 0.994

EPOCH 546 ...
Validation Accuracy = 0.994

EPOCH 547 ...
Validation Accuracy = 0.994

EPOCH 548 ...
Validation Accuracy = 0.994

EPOCH 549 ...
Validation Accuracy = 0.994

EPOCH 550 ...
Validation Accuracy = 0.994

EPOCH 551 ...
Validation Accuracy = 0.994

EPOCH 552 ...
Validation Accuracy = 0.994

EPOCH 553 ...
Validation Accuracy = 0.994

EPOCH 554 ...
Validation Accuracy = 0.994

EPOCH 555 ...
Validation Accuracy = 0.994

EPOCH 556 ...
Validation Accuracy = 0.994

EPOCH 557 ...
Validation Accuracy = 0.994

EPOCH 558 ...
Validation Accuracy = 0.994

EPOCH 559 ...
Validation Accuracy = 0.994

EPOCH 560 ...
Validation Accuracy = 0.994

EPOCH 561 ...
Validation Accuracy = 0.994

EPOCH 562 ...
Validation Accuracy = 0.994

EPOCH 563 ...
Validation Accuracy = 0.994

EPOCH 564 ...
Validation Accuracy = 0.994

EPOCH 565 ...
Validation Accuracy = 0.994

EPOCH 566 ...
Validation Accuracy = 0.994

EPOCH 567 ...
Validation Accuracy = 0.994

EPOCH 568 ...
Validation Accuracy = 0.994

EPOCH 569 ...
Validation Accuracy = 0.994

EPOCH 570 ...
Validation Accuracy = 0.994

EPOCH 571 ...
Validation Accuracy = 0.994

EPOCH 572 ...
Validation Accuracy = 0.994

EPOCH 573 ...
Validation Accuracy = 0.994

EPOCH 574 ...
Validation Accuracy = 0.994

EPOCH 575 ...
Validation Accuracy = 0.994

EPOCH 576 ...
Validation Accuracy = 0.994

EPOCH 577 ...
Validation Accuracy = 0.994

EPOCH 578 ...
Validation Accuracy = 0.994

EPOCH 579 ...
Validation Accuracy = 0.994

EPOCH 580 ...
Validation Accuracy = 0.994

EPOCH 581 ...
Validation Accuracy = 0.994

EPOCH 582 ...
Validation Accuracy = 0.994

EPOCH 583 ...
Validation Accuracy = 0.994

EPOCH 584 ...
Validation Accuracy = 0.994

EPOCH 585 ...
Validation Accuracy = 0.994

EPOCH 586 ...
Validation Accuracy = 0.994

EPOCH 587 ...
Validation Accuracy = 0.994

EPOCH 588 ...
Validation Accuracy = 0.994

EPOCH 589 ...
Validation Accuracy = 0.994

EPOCH 590 ...
Validation Accuracy = 0.994

EPOCH 591 ...
Validation Accuracy = 0.994

EPOCH 592 ...
Validation Accuracy = 0.994

EPOCH 593 ...
Validation Accuracy = 0.994

EPOCH 594 ...
Validation Accuracy = 0.994

EPOCH 595 ...
Validation Accuracy = 0.994

EPOCH 596 ...
Validation Accuracy = 0.994

EPOCH 597 ...
Validation Accuracy = 0.987

EPOCH 598 ...
Validation Accuracy = 0.993

EPOCH 599 ...
Validation Accuracy = 0.994

EPOCH 600 ...
Validation Accuracy = 0.994

EPOCH 601 ...
Validation Accuracy = 0.994

EPOCH 602 ...
Validation Accuracy = 0.994

EPOCH 603 ...
Validation Accuracy = 0.994

EPOCH 604 ...
Validation Accuracy = 0.994

EPOCH 605 ...
Validation Accuracy = 0.994

EPOCH 606 ...
Validation Accuracy = 0.994

EPOCH 607 ...
Validation Accuracy = 0.994

EPOCH 608 ...
Validation Accuracy = 0.994

EPOCH 609 ...
Validation Accuracy = 0.994

EPOCH 610 ...
Validation Accuracy = 0.994

EPOCH 611 ...
Validation Accuracy = 0.994

EPOCH 612 ...
Validation Accuracy = 0.994

EPOCH 613 ...
Validation Accuracy = 0.994

EPOCH 614 ...
Validation Accuracy = 0.994

EPOCH 615 ...
Validation Accuracy = 0.994

EPOCH 616 ...
Validation Accuracy = 0.994

EPOCH 617 ...
Validation Accuracy = 0.994

EPOCH 618 ...
Validation Accuracy = 0.994

EPOCH 619 ...
Validation Accuracy = 0.994

EPOCH 620 ...
Validation Accuracy = 0.994

EPOCH 621 ...
Validation Accuracy = 0.994

EPOCH 622 ...
Validation Accuracy = 0.994

EPOCH 623 ...
Validation Accuracy = 0.994

EPOCH 624 ...
Validation Accuracy = 0.994

EPOCH 625 ...
Validation Accuracy = 0.994

EPOCH 626 ...
Validation Accuracy = 0.994

EPOCH 627 ...
Validation Accuracy = 0.994

EPOCH 628 ...
Validation Accuracy = 0.994

EPOCH 629 ...
Validation Accuracy = 0.994

EPOCH 630 ...
Validation Accuracy = 0.994

EPOCH 631 ...
Validation Accuracy = 0.994

EPOCH 632 ...
Validation Accuracy = 0.994

EPOCH 633 ...
Validation Accuracy = 0.994

EPOCH 634 ...
Validation Accuracy = 0.994

EPOCH 635 ...
Validation Accuracy = 0.994

EPOCH 636 ...
Validation Accuracy = 0.994

EPOCH 637 ...
Validation Accuracy = 0.994

EPOCH 638 ...
Validation Accuracy = 0.994

EPOCH 639 ...
Validation Accuracy = 0.994

EPOCH 640 ...
Validation Accuracy = 0.994

EPOCH 641 ...
Validation Accuracy = 0.994

EPOCH 642 ...
Validation Accuracy = 0.994

EPOCH 643 ...
Validation Accuracy = 0.994

EPOCH 644 ...
Validation Accuracy = 0.994

EPOCH 645 ...
Validation Accuracy = 0.994

EPOCH 646 ...
Validation Accuracy = 0.994

EPOCH 647 ...
Validation Accuracy = 0.994

EPOCH 648 ...
Validation Accuracy = 0.994

EPOCH 649 ...
Validation Accuracy = 0.994

EPOCH 650 ...
Validation Accuracy = 0.994

EPOCH 651 ...
Validation Accuracy = 0.994

EPOCH 652 ...
Validation Accuracy = 0.994

EPOCH 653 ...
Validation Accuracy = 0.994

EPOCH 654 ...
Validation Accuracy = 0.994

EPOCH 655 ...
Validation Accuracy = 0.994

EPOCH 656 ...
Validation Accuracy = 0.994

EPOCH 657 ...
Validation Accuracy = 0.994

EPOCH 658 ...
Validation Accuracy = 0.994

EPOCH 659 ...
Validation Accuracy = 0.994

EPOCH 660 ...
Validation Accuracy = 0.994

EPOCH 661 ...
Validation Accuracy = 0.994

EPOCH 662 ...
Validation Accuracy = 0.994

EPOCH 663 ...
Validation Accuracy = 0.994

EPOCH 664 ...
Validation Accuracy = 0.994

EPOCH 665 ...
Validation Accuracy = 0.994

EPOCH 666 ...
Validation Accuracy = 0.994

EPOCH 667 ...
Validation Accuracy = 0.994

EPOCH 668 ...
Validation Accuracy = 0.994

EPOCH 669 ...
Validation Accuracy = 0.987

EPOCH 670 ...
Validation Accuracy = 0.990

EPOCH 671 ...
Validation Accuracy = 0.994

EPOCH 672 ...
Validation Accuracy = 0.994

EPOCH 673 ...
Validation Accuracy = 0.994

EPOCH 674 ...
Validation Accuracy = 0.995

EPOCH 675 ...
Validation Accuracy = 0.995

EPOCH 676 ...
Validation Accuracy = 0.995

EPOCH 677 ...
Validation Accuracy = 0.995

EPOCH 678 ...
Validation Accuracy = 0.995

EPOCH 679 ...
Validation Accuracy = 0.995

EPOCH 680 ...
Validation Accuracy = 0.994

EPOCH 681 ...
Validation Accuracy = 0.994

EPOCH 682 ...
Validation Accuracy = 0.994

EPOCH 683 ...
Validation Accuracy = 0.995

EPOCH 684 ...
Validation Accuracy = 0.994

EPOCH 685 ...
Validation Accuracy = 0.995

EPOCH 686 ...
Validation Accuracy = 0.994

EPOCH 687 ...
Validation Accuracy = 0.995

EPOCH 688 ...
Validation Accuracy = 0.995

EPOCH 689 ...
Validation Accuracy = 0.995

EPOCH 690 ...
Validation Accuracy = 0.995

EPOCH 691 ...
Validation Accuracy = 0.995

EPOCH 692 ...
Validation Accuracy = 0.995

EPOCH 693 ...
Validation Accuracy = 0.995

EPOCH 694 ...
Validation Accuracy = 0.995

EPOCH 695 ...
Validation Accuracy = 0.995

EPOCH 696 ...
Validation Accuracy = 0.995

EPOCH 697 ...
Validation Accuracy = 0.995

EPOCH 698 ...
Validation Accuracy = 0.995

EPOCH 699 ...
Validation Accuracy = 0.995

EPOCH 700 ...
Validation Accuracy = 0.995

EPOCH 701 ...
Validation Accuracy = 0.995

EPOCH 702 ...
Validation Accuracy = 0.995

EPOCH 703 ...
Validation Accuracy = 0.995

EPOCH 704 ...
Validation Accuracy = 0.995

EPOCH 705 ...
Validation Accuracy = 0.995

EPOCH 706 ...
Validation Accuracy = 0.995

EPOCH 707 ...
Validation Accuracy = 0.995

EPOCH 708 ...
Validation Accuracy = 0.995

EPOCH 709 ...
Validation Accuracy = 0.995

EPOCH 710 ...
Validation Accuracy = 0.995

EPOCH 711 ...
Validation Accuracy = 0.995

EPOCH 712 ...
Validation Accuracy = 0.995

EPOCH 713 ...
Validation Accuracy = 0.995

EPOCH 714 ...
Validation Accuracy = 0.995

EPOCH 715 ...
Validation Accuracy = 0.995

EPOCH 716 ...
Validation Accuracy = 0.995

EPOCH 717 ...
Validation Accuracy = 0.995

EPOCH 718 ...
Validation Accuracy = 0.995

EPOCH 719 ...
Validation Accuracy = 0.995

EPOCH 720 ...
Validation Accuracy = 0.995

EPOCH 721 ...
Validation Accuracy = 0.995

EPOCH 722 ...
Validation Accuracy = 0.995

EPOCH 723 ...
Validation Accuracy = 0.995

EPOCH 724 ...
Validation Accuracy = 0.995

EPOCH 725 ...
Validation Accuracy = 0.995

EPOCH 726 ...
Validation Accuracy = 0.995

EPOCH 727 ...
Validation Accuracy = 0.995

EPOCH 728 ...
Validation Accuracy = 0.995

EPOCH 729 ...
Validation Accuracy = 0.995

EPOCH 730 ...
Validation Accuracy = 0.995

EPOCH 731 ...
Validation Accuracy = 0.994

EPOCH 732 ...
Validation Accuracy = 0.994

EPOCH 733 ...
Validation Accuracy = 0.995

EPOCH 734 ...
Validation Accuracy = 0.994

EPOCH 735 ...
Validation Accuracy = 0.994

EPOCH 736 ...
Validation Accuracy = 0.994

EPOCH 737 ...
Validation Accuracy = 0.995

EPOCH 738 ...
Validation Accuracy = 0.994

EPOCH 739 ...
Validation Accuracy = 0.990

EPOCH 740 ...
Validation Accuracy = 0.991

EPOCH 741 ...
Validation Accuracy = 0.994

EPOCH 742 ...
Validation Accuracy = 0.994

EPOCH 743 ...
Validation Accuracy = 0.994

EPOCH 744 ...
Validation Accuracy = 0.994

EPOCH 745 ...
Validation Accuracy = 0.995

EPOCH 746 ...
Validation Accuracy = 0.995

EPOCH 747 ...
Validation Accuracy = 0.995

EPOCH 748 ...
Validation Accuracy = 0.995

EPOCH 749 ...
Validation Accuracy = 0.995

EPOCH 750 ...
Validation Accuracy = 0.995

EPOCH 751 ...
Validation Accuracy = 0.995

EPOCH 752 ...
Validation Accuracy = 0.995

EPOCH 753 ...
Validation Accuracy = 0.995

EPOCH 754 ...
Validation Accuracy = 0.995

EPOCH 755 ...
Validation Accuracy = 0.995

EPOCH 756 ...
Validation Accuracy = 0.995

EPOCH 757 ...
Validation Accuracy = 0.995

EPOCH 758 ...
Validation Accuracy = 0.995

EPOCH 759 ...
Validation Accuracy = 0.995

EPOCH 760 ...
Validation Accuracy = 0.995

EPOCH 761 ...
Validation Accuracy = 0.995

EPOCH 762 ...
Validation Accuracy = 0.995

EPOCH 763 ...
Validation Accuracy = 0.995

EPOCH 764 ...
Validation Accuracy = 0.995

EPOCH 765 ...
Validation Accuracy = 0.995

EPOCH 766 ...
Validation Accuracy = 0.995

EPOCH 767 ...
Validation Accuracy = 0.995

EPOCH 768 ...
Validation Accuracy = 0.995

EPOCH 769 ...
Validation Accuracy = 0.995

EPOCH 770 ...
Validation Accuracy = 0.995

EPOCH 771 ...
Validation Accuracy = 0.995

EPOCH 772 ...
Validation Accuracy = 0.995

EPOCH 773 ...
Validation Accuracy = 0.995

EPOCH 774 ...
Validation Accuracy = 0.995

EPOCH 775 ...
Validation Accuracy = 0.995

EPOCH 776 ...
Validation Accuracy = 0.995

EPOCH 777 ...
Validation Accuracy = 0.995

EPOCH 778 ...
Validation Accuracy = 0.995

EPOCH 779 ...
Validation Accuracy = 0.995

EPOCH 780 ...
Validation Accuracy = 0.995

EPOCH 781 ...
Validation Accuracy = 0.995

EPOCH 782 ...
Validation Accuracy = 0.995

EPOCH 783 ...
Validation Accuracy = 0.995

EPOCH 784 ...
Validation Accuracy = 0.995

EPOCH 785 ...
Validation Accuracy = 0.995

EPOCH 786 ...
Validation Accuracy = 0.995

EPOCH 787 ...
Validation Accuracy = 0.995

EPOCH 788 ...
Validation Accuracy = 0.995

EPOCH 789 ...
Validation Accuracy = 0.995

EPOCH 790 ...
Validation Accuracy = 0.995

EPOCH 791 ...
Validation Accuracy = 0.995

EPOCH 792 ...
Validation Accuracy = 0.995

EPOCH 793 ...
Validation Accuracy = 0.995

EPOCH 794 ...
Validation Accuracy = 0.995

EPOCH 795 ...
Validation Accuracy = 0.995

EPOCH 796 ...
Validation Accuracy = 0.995

EPOCH 797 ...
Validation Accuracy = 0.995

EPOCH 798 ...
Validation Accuracy = 0.995

EPOCH 799 ...
Validation Accuracy = 0.995

EPOCH 800 ...
Validation Accuracy = 0.995

EPOCH 801 ...
Validation Accuracy = 0.995

EPOCH 802 ...
Validation Accuracy = 0.995

EPOCH 803 ...
Validation Accuracy = 0.995

EPOCH 804 ...
Validation Accuracy = 0.995

EPOCH 805 ...
Validation Accuracy = 0.995

EPOCH 806 ...
Validation Accuracy = 0.995

EPOCH 807 ...
Validation Accuracy = 0.995

EPOCH 808 ...
Validation Accuracy = 0.995

EPOCH 809 ...
Validation Accuracy = 0.995

EPOCH 810 ...
Validation Accuracy = 0.995

EPOCH 811 ...
Validation Accuracy = 0.995

EPOCH 812 ...
Validation Accuracy = 0.995

EPOCH 813 ...
Validation Accuracy = 0.995

EPOCH 814 ...
Validation Accuracy = 0.995

EPOCH 815 ...
Validation Accuracy = 0.987

EPOCH 816 ...
Validation Accuracy = 0.992

EPOCH 817 ...
Validation Accuracy = 0.994

EPOCH 818 ...
Validation Accuracy = 0.995

EPOCH 819 ...
Validation Accuracy = 0.995

EPOCH 820 ...
Validation Accuracy = 0.995

EPOCH 821 ...
Validation Accuracy = 0.995

EPOCH 822 ...
Validation Accuracy = 0.995

EPOCH 823 ...
Validation Accuracy = 0.995

EPOCH 824 ...
Validation Accuracy = 0.995

EPOCH 825 ...
Validation Accuracy = 0.995

EPOCH 826 ...
Validation Accuracy = 0.995

EPOCH 827 ...
Validation Accuracy = 0.995

EPOCH 828 ...
Validation Accuracy = 0.995

EPOCH 829 ...
Validation Accuracy = 0.995

EPOCH 830 ...
Validation Accuracy = 0.995

EPOCH 831 ...
Validation Accuracy = 0.995

EPOCH 832 ...
Validation Accuracy = 0.995

EPOCH 833 ...
Validation Accuracy = 0.995

EPOCH 834 ...
Validation Accuracy = 0.995

EPOCH 835 ...
Validation Accuracy = 0.995

EPOCH 836 ...
Validation Accuracy = 0.995

EPOCH 837 ...
Validation Accuracy = 0.995

EPOCH 838 ...
Validation Accuracy = 0.995

EPOCH 839 ...
Validation Accuracy = 0.995

EPOCH 840 ...
Validation Accuracy = 0.995

EPOCH 841 ...
Validation Accuracy = 0.995

EPOCH 842 ...
Validation Accuracy = 0.995

EPOCH 843 ...
Validation Accuracy = 0.995

EPOCH 844 ...
Validation Accuracy = 0.995

EPOCH 845 ...
Validation Accuracy = 0.995

EPOCH 846 ...
Validation Accuracy = 0.995

EPOCH 847 ...
Validation Accuracy = 0.995

EPOCH 848 ...
Validation Accuracy = 0.995

EPOCH 849 ...
Validation Accuracy = 0.995

EPOCH 850 ...
Validation Accuracy = 0.995

EPOCH 851 ...
Validation Accuracy = 0.995

EPOCH 852 ...
Validation Accuracy = 0.995

EPOCH 853 ...
Validation Accuracy = 0.995

EPOCH 854 ...
Validation Accuracy = 0.995

EPOCH 855 ...
Validation Accuracy = 0.995

EPOCH 856 ...
Validation Accuracy = 0.995

EPOCH 857 ...
Validation Accuracy = 0.995

EPOCH 858 ...
Validation Accuracy = 0.995

EPOCH 859 ...
Validation Accuracy = 0.995

EPOCH 860 ...
Validation Accuracy = 0.995

EPOCH 861 ...
Validation Accuracy = 0.995

EPOCH 862 ...
Validation Accuracy = 0.995

EPOCH 863 ...
Validation Accuracy = 0.995

EPOCH 864 ...
Validation Accuracy = 0.995

EPOCH 865 ...
Validation Accuracy = 0.995

EPOCH 866 ...
Validation Accuracy = 0.995

EPOCH 867 ...
Validation Accuracy = 0.995

EPOCH 868 ...
Validation Accuracy = 0.995

EPOCH 869 ...
Validation Accuracy = 0.995

EPOCH 870 ...
Validation Accuracy = 0.995

EPOCH 871 ...
Validation Accuracy = 0.995

EPOCH 872 ...
Validation Accuracy = 0.995

EPOCH 873 ...
Validation Accuracy = 0.995

EPOCH 874 ...
Validation Accuracy = 0.995

EPOCH 875 ...
Validation Accuracy = 0.995

EPOCH 876 ...
Validation Accuracy = 0.995

EPOCH 877 ...
Validation Accuracy = 0.995

EPOCH 878 ...
Validation Accuracy = 0.995

EPOCH 879 ...
Validation Accuracy = 0.995

EPOCH 880 ...
Validation Accuracy = 0.995

EPOCH 881 ...
Validation Accuracy = 0.995

EPOCH 882 ...
Validation Accuracy = 0.995

EPOCH 883 ...
Validation Accuracy = 0.995

EPOCH 884 ...
Validation Accuracy = 0.995

EPOCH 885 ...
Validation Accuracy = 0.995

EPOCH 886 ...
Validation Accuracy = 0.995

EPOCH 887 ...
Validation Accuracy = 0.995

EPOCH 888 ...
Validation Accuracy = 0.995

EPOCH 889 ...
Validation Accuracy = 0.995

EPOCH 890 ...
Validation Accuracy = 0.995

EPOCH 891 ...
Validation Accuracy = 0.995

EPOCH 892 ...
Validation Accuracy = 0.995

EPOCH 893 ...
Validation Accuracy = 0.995

EPOCH 894 ...
Validation Accuracy = 0.995

EPOCH 895 ...
Validation Accuracy = 0.995

EPOCH 896 ...
Validation Accuracy = 0.995

EPOCH 897 ...
Validation Accuracy = 0.986

EPOCH 898 ...
Validation Accuracy = 0.993

EPOCH 899 ...
Validation Accuracy = 0.994

EPOCH 900 ...
Validation Accuracy = 0.994

EPOCH 901 ...
Validation Accuracy = 0.994

EPOCH 902 ...
Validation Accuracy = 0.994

EPOCH 903 ...
Validation Accuracy = 0.995

EPOCH 904 ...
Validation Accuracy = 0.995

EPOCH 905 ...
Validation Accuracy = 0.995

EPOCH 906 ...
Validation Accuracy = 0.995

EPOCH 907 ...
Validation Accuracy = 0.995

EPOCH 908 ...
Validation Accuracy = 0.995

EPOCH 909 ...
Validation Accuracy = 0.995

EPOCH 910 ...
Validation Accuracy = 0.995

EPOCH 911 ...
Validation Accuracy = 0.995

EPOCH 912 ...
Validation Accuracy = 0.995

EPOCH 913 ...
Validation Accuracy = 0.995

EPOCH 914 ...
Validation Accuracy = 0.995

EPOCH 915 ...
Validation Accuracy = 0.995

EPOCH 916 ...
Validation Accuracy = 0.995

EPOCH 917 ...
Validation Accuracy = 0.995

EPOCH 918 ...
Validation Accuracy = 0.995

EPOCH 919 ...
Validation Accuracy = 0.995

EPOCH 920 ...
Validation Accuracy = 0.995

EPOCH 921 ...
Validation Accuracy = 0.995

EPOCH 922 ...
Validation Accuracy = 0.995

EPOCH 923 ...
Validation Accuracy = 0.995

EPOCH 924 ...
Validation Accuracy = 0.995

EPOCH 925 ...
Validation Accuracy = 0.995

EPOCH 926 ...
Validation Accuracy = 0.995

EPOCH 927 ...
Validation Accuracy = 0.995

EPOCH 928 ...
Validation Accuracy = 0.995

EPOCH 929 ...
Validation Accuracy = 0.995

EPOCH 930 ...
Validation Accuracy = 0.995

EPOCH 931 ...
Validation Accuracy = 0.995

EPOCH 932 ...
Validation Accuracy = 0.995

EPOCH 933 ...
Validation Accuracy = 0.995

EPOCH 934 ...
Validation Accuracy = 0.995

EPOCH 935 ...
Validation Accuracy = 0.995

EPOCH 936 ...
Validation Accuracy = 0.995

EPOCH 937 ...
Validation Accuracy = 0.995

EPOCH 938 ...
Validation Accuracy = 0.995

EPOCH 939 ...
Validation Accuracy = 0.995

EPOCH 940 ...
Validation Accuracy = 0.995

EPOCH 941 ...
Validation Accuracy = 0.995

EPOCH 942 ...
Validation Accuracy = 0.995

EPOCH 943 ...
Validation Accuracy = 0.995

EPOCH 944 ...
Validation Accuracy = 0.995

EPOCH 945 ...
Validation Accuracy = 0.995

EPOCH 946 ...
Validation Accuracy = 0.995

EPOCH 947 ...
Validation Accuracy = 0.995

EPOCH 948 ...
Validation Accuracy = 0.995

EPOCH 949 ...
Validation Accuracy = 0.995

EPOCH 950 ...
Validation Accuracy = 0.995

EPOCH 951 ...
Validation Accuracy = 0.995

EPOCH 952 ...
Validation Accuracy = 0.995

EPOCH 953 ...
Validation Accuracy = 0.995

EPOCH 954 ...
Validation Accuracy = 0.995

EPOCH 955 ...
Validation Accuracy = 0.995

EPOCH 956 ...
Validation Accuracy = 0.995

EPOCH 957 ...
Validation Accuracy = 0.995

EPOCH 958 ...
Validation Accuracy = 0.995

EPOCH 959 ...
Validation Accuracy = 0.995

EPOCH 960 ...
Validation Accuracy = 0.995

EPOCH 961 ...
Validation Accuracy = 0.995

EPOCH 962 ...
Validation Accuracy = 0.995

EPOCH 963 ...
Validation Accuracy = 0.995

EPOCH 964 ...
Validation Accuracy = 0.995

EPOCH 965 ...
Validation Accuracy = 0.995

EPOCH 966 ...
Validation Accuracy = 0.995

EPOCH 967 ...
Validation Accuracy = 0.995

EPOCH 968 ...
Validation Accuracy = 0.995

EPOCH 969 ...
Validation Accuracy = 0.995

EPOCH 970 ...
Validation Accuracy = 0.995

EPOCH 971 ...
Validation Accuracy = 0.995

EPOCH 972 ...
Validation Accuracy = 0.995

EPOCH 973 ...
Validation Accuracy = 0.995

EPOCH 974 ...
Validation Accuracy = 0.995

EPOCH 975 ...
Validation Accuracy = 0.995

EPOCH 976 ...
Validation Accuracy = 0.995

EPOCH 977 ...
Validation Accuracy = 0.995

EPOCH 978 ...
Validation Accuracy = 0.995

EPOCH 979 ...
Validation Accuracy = 0.995

EPOCH 980 ...
Validation Accuracy = 0.995

EPOCH 981 ...
Validation Accuracy = 0.995

EPOCH 982 ...
Validation Accuracy = 0.995

EPOCH 983 ...
Validation Accuracy = 0.995

EPOCH 984 ...
Validation Accuracy = 0.995

EPOCH 985 ...
Validation Accuracy = 0.995

EPOCH 986 ...
Validation Accuracy = 0.995

EPOCH 987 ...
Validation Accuracy = 0.984

EPOCH 988 ...
Validation Accuracy = 0.992

EPOCH 989 ...
Validation Accuracy = 0.994

EPOCH 990 ...
Validation Accuracy = 0.995

EPOCH 991 ...
Validation Accuracy = 0.995

EPOCH 992 ...
Validation Accuracy = 0.995

EPOCH 993 ...
Validation Accuracy = 0.995

EPOCH 994 ...
Validation Accuracy = 0.995

EPOCH 995 ...
Validation Accuracy = 0.995

EPOCH 996 ...
Validation Accuracy = 0.995

EPOCH 997 ...
Validation Accuracy = 0.995

EPOCH 998 ...
Validation Accuracy = 0.995

EPOCH 999 ...
Validation Accuracy = 0.995

EPOCH 1000 ...
Validation Accuracy = 0.995

EPOCH 1001 ...
Validation Accuracy = 0.995

EPOCH 1002 ...
Validation Accuracy = 0.995

EPOCH 1003 ...
Validation Accuracy = 0.995

EPOCH 1004 ...
Validation Accuracy = 0.995

EPOCH 1005 ...
Validation Accuracy = 0.995

EPOCH 1006 ...
Validation Accuracy = 0.995

EPOCH 1007 ...
Validation Accuracy = 0.995

EPOCH 1008 ...
Validation Accuracy = 0.995

EPOCH 1009 ...
Validation Accuracy = 0.995

EPOCH 1010 ...
Validation Accuracy = 0.995

EPOCH 1011 ...
Validation Accuracy = 0.995

EPOCH 1012 ...
Validation Accuracy = 0.995

EPOCH 1013 ...
Validation Accuracy = 0.995

EPOCH 1014 ...
Validation Accuracy = 0.995

EPOCH 1015 ...
Validation Accuracy = 0.995

EPOCH 1016 ...
Validation Accuracy = 0.995

EPOCH 1017 ...
Validation Accuracy = 0.995

EPOCH 1018 ...
Validation Accuracy = 0.995

EPOCH 1019 ...
Validation Accuracy = 0.995

EPOCH 1020 ...
Validation Accuracy = 0.995

EPOCH 1021 ...
Validation Accuracy = 0.995

EPOCH 1022 ...
Validation Accuracy = 0.995

EPOCH 1023 ...
Validation Accuracy = 0.995

EPOCH 1024 ...
Validation Accuracy = 0.995

EPOCH 1025 ...
Validation Accuracy = 0.995

EPOCH 1026 ...
Validation Accuracy = 0.995

EPOCH 1027 ...
Validation Accuracy = 0.995

EPOCH 1028 ...
Validation Accuracy = 0.995

EPOCH 1029 ...
Validation Accuracy = 0.995

EPOCH 1030 ...
Validation Accuracy = 0.995

EPOCH 1031 ...
Validation Accuracy = 0.995

EPOCH 1032 ...
Validation Accuracy = 0.995

EPOCH 1033 ...
Validation Accuracy = 0.995

EPOCH 1034 ...
Validation Accuracy = 0.995

EPOCH 1035 ...
Validation Accuracy = 0.995

EPOCH 1036 ...
Validation Accuracy = 0.995

EPOCH 1037 ...
Validation Accuracy = 0.995

EPOCH 1038 ...
Validation Accuracy = 0.995

EPOCH 1039 ...
Validation Accuracy = 0.995

EPOCH 1040 ...
Validation Accuracy = 0.995

EPOCH 1041 ...
Validation Accuracy = 0.995

EPOCH 1042 ...
Validation Accuracy = 0.995

EPOCH 1043 ...
Validation Accuracy = 0.995

EPOCH 1044 ...
Validation Accuracy = 0.995

EPOCH 1045 ...
Validation Accuracy = 0.995

EPOCH 1046 ...
Validation Accuracy = 0.995

EPOCH 1047 ...
Validation Accuracy = 0.995

EPOCH 1048 ...
Validation Accuracy = 0.995

EPOCH 1049 ...
Validation Accuracy = 0.995

EPOCH 1050 ...
Validation Accuracy = 0.995

EPOCH 1051 ...
Validation Accuracy = 0.995

EPOCH 1052 ...
Validation Accuracy = 0.995

EPOCH 1053 ...
Validation Accuracy = 0.995

EPOCH 1054 ...
Validation Accuracy = 0.995

EPOCH 1055 ...
Validation Accuracy = 0.995

EPOCH 1056 ...
Validation Accuracy = 0.995

EPOCH 1057 ...
Validation Accuracy = 0.995

EPOCH 1058 ...
Validation Accuracy = 0.995

EPOCH 1059 ...
Validation Accuracy = 0.995

EPOCH 1060 ...
Validation Accuracy = 0.995

EPOCH 1061 ...
Validation Accuracy = 0.995

EPOCH 1062 ...
Validation Accuracy = 0.995

EPOCH 1063 ...
Validation Accuracy = 0.995

EPOCH 1064 ...
Validation Accuracy = 0.995

EPOCH 1065 ...
Validation Accuracy = 0.995

EPOCH 1066 ...
Validation Accuracy = 0.995

EPOCH 1067 ...
Validation Accuracy = 0.995

EPOCH 1068 ...
Validation Accuracy = 0.995

EPOCH 1069 ...
Validation Accuracy = 0.995

EPOCH 1070 ...
Validation Accuracy = 0.995

EPOCH 1071 ...
Validation Accuracy = 0.995

EPOCH 1072 ...
Validation Accuracy = 0.995

EPOCH 1073 ...
Validation Accuracy = 0.995

EPOCH 1074 ...
Validation Accuracy = 0.995

EPOCH 1075 ...
Validation Accuracy = 0.995

EPOCH 1076 ...
Validation Accuracy = 0.995

EPOCH 1077 ...
Validation Accuracy = 0.995

EPOCH 1078 ...
Validation Accuracy = 0.995

EPOCH 1079 ...
Validation Accuracy = 0.995

EPOCH 1080 ...
Validation Accuracy = 0.991

EPOCH 1081 ...
Validation Accuracy = 0.993

EPOCH 1082 ...
Validation Accuracy = 0.995

EPOCH 1083 ...
Validation Accuracy = 0.995

EPOCH 1084 ...
Validation Accuracy = 0.995

EPOCH 1085 ...
Validation Accuracy = 0.995

EPOCH 1086 ...
Validation Accuracy = 0.995

EPOCH 1087 ...
Validation Accuracy = 0.995

EPOCH 1088 ...
Validation Accuracy = 0.995

EPOCH 1089 ...
Validation Accuracy = 0.995

EPOCH 1090 ...
Validation Accuracy = 0.995

EPOCH 1091 ...
Validation Accuracy = 0.995

EPOCH 1092 ...
Validation Accuracy = 0.995

EPOCH 1093 ...
Validation Accuracy = 0.995

EPOCH 1094 ...
Validation Accuracy = 0.995

EPOCH 1095 ...
Validation Accuracy = 0.995

EPOCH 1096 ...
Validation Accuracy = 0.995

EPOCH 1097 ...
Validation Accuracy = 0.995

EPOCH 1098 ...
Validation Accuracy = 0.995

EPOCH 1099 ...
Validation Accuracy = 0.995

EPOCH 1100 ...
Validation Accuracy = 0.995

EPOCH 1101 ...
Validation Accuracy = 0.995

EPOCH 1102 ...
Validation Accuracy = 0.995

EPOCH 1103 ...
Validation Accuracy = 0.995

EPOCH 1104 ...
Validation Accuracy = 0.995

EPOCH 1105 ...
Validation Accuracy = 0.995

EPOCH 1106 ...
Validation Accuracy = 0.995

EPOCH 1107 ...
Validation Accuracy = 0.995

EPOCH 1108 ...
Validation Accuracy = 0.995

EPOCH 1109 ...
Validation Accuracy = 0.995

EPOCH 1110 ...
Validation Accuracy = 0.995

EPOCH 1111 ...
Validation Accuracy = 0.995

EPOCH 1112 ...
Validation Accuracy = 0.995

EPOCH 1113 ...
Validation Accuracy = 0.995

EPOCH 1114 ...
Validation Accuracy = 0.995

EPOCH 1115 ...
Validation Accuracy = 0.995

EPOCH 1116 ...
Validation Accuracy = 0.995

EPOCH 1117 ...
Validation Accuracy = 0.995

EPOCH 1118 ...
Validation Accuracy = 0.995

EPOCH 1119 ...
Validation Accuracy = 0.995

EPOCH 1120 ...
Validation Accuracy = 0.995

EPOCH 1121 ...
Validation Accuracy = 0.995

EPOCH 1122 ...
Validation Accuracy = 0.995

EPOCH 1123 ...
Validation Accuracy = 0.995

EPOCH 1124 ...
Validation Accuracy = 0.995

EPOCH 1125 ...
Validation Accuracy = 0.995

EPOCH 1126 ...
Validation Accuracy = 0.995

EPOCH 1127 ...
Validation Accuracy = 0.995

EPOCH 1128 ...
Validation Accuracy = 0.995

EPOCH 1129 ...
Validation Accuracy = 0.995

EPOCH 1130 ...
Validation Accuracy = 0.995

EPOCH 1131 ...
Validation Accuracy = 0.995

EPOCH 1132 ...
Validation Accuracy = 0.995

EPOCH 1133 ...
Validation Accuracy = 0.995

EPOCH 1134 ...
Validation Accuracy = 0.995

EPOCH 1135 ...
Validation Accuracy = 0.995

EPOCH 1136 ...
Validation Accuracy = 0.995

EPOCH 1137 ...
Validation Accuracy = 0.995

EPOCH 1138 ...
Validation Accuracy = 0.995

EPOCH 1139 ...
Validation Accuracy = 0.995

EPOCH 1140 ...
Validation Accuracy = 0.995

EPOCH 1141 ...
Validation Accuracy = 0.995

EPOCH 1142 ...
Validation Accuracy = 0.995

EPOCH 1143 ...
Validation Accuracy = 0.995

EPOCH 1144 ...
Validation Accuracy = 0.995

EPOCH 1145 ...
Validation Accuracy = 0.995

EPOCH 1146 ...
Validation Accuracy = 0.995

EPOCH 1147 ...
Validation Accuracy = 0.995

EPOCH 1148 ...
Validation Accuracy = 0.995

EPOCH 1149 ...
Validation Accuracy = 0.995

EPOCH 1150 ...
Validation Accuracy = 0.995

EPOCH 1151 ...
Validation Accuracy = 0.995

EPOCH 1152 ...
Validation Accuracy = 0.995

EPOCH 1153 ...
Validation Accuracy = 0.995

EPOCH 1154 ...
Validation Accuracy = 0.995

EPOCH 1155 ...
Validation Accuracy = 0.995

EPOCH 1156 ...
Validation Accuracy = 0.995

EPOCH 1157 ...
Validation Accuracy = 0.995

EPOCH 1158 ...
Validation Accuracy = 0.995

EPOCH 1159 ...
Validation Accuracy = 0.995

EPOCH 1160 ...
Validation Accuracy = 0.995

EPOCH 1161 ...
Validation Accuracy = 0.995

EPOCH 1162 ...
Validation Accuracy = 0.995

EPOCH 1163 ...
Validation Accuracy = 0.995

EPOCH 1164 ...
Validation Accuracy = 0.995

EPOCH 1165 ...
Validation Accuracy = 0.995

EPOCH 1166 ...
Validation Accuracy = 0.995

EPOCH 1167 ...
Validation Accuracy = 0.995

EPOCH 1168 ...
Validation Accuracy = 0.995

EPOCH 1169 ...
Validation Accuracy = 0.995

EPOCH 1170 ...
Validation Accuracy = 0.995

EPOCH 1171 ...
Validation Accuracy = 0.995

EPOCH 1172 ...
Validation Accuracy = 0.995

EPOCH 1173 ...
Validation Accuracy = 0.995

EPOCH 1174 ...
Validation Accuracy = 0.995

EPOCH 1175 ...
Validation Accuracy = 0.995

EPOCH 1176 ...
Validation Accuracy = 0.995

EPOCH 1177 ...
Validation Accuracy = 0.995

EPOCH 1178 ...
Validation Accuracy = 0.995

EPOCH 1179 ...
Validation Accuracy = 0.995

EPOCH 1180 ...
Validation Accuracy = 0.995

EPOCH 1181 ...
Validation Accuracy = 0.995

EPOCH 1182 ...
Validation Accuracy = 0.995

EPOCH 1183 ...
Validation Accuracy = 0.995

EPOCH 1184 ...
Validation Accuracy = 0.995

EPOCH 1185 ...
Validation Accuracy = 0.995

EPOCH 1186 ...
Validation Accuracy = 0.995

EPOCH 1187 ...
Validation Accuracy = 0.995

EPOCH 1188 ...
Validation Accuracy = 0.995

EPOCH 1189 ...
Validation Accuracy = 0.995

EPOCH 1190 ...
Validation Accuracy = 0.995

EPOCH 1191 ...
Validation Accuracy = 0.995

EPOCH 1192 ...
Validation Accuracy = 0.995

EPOCH 1193 ...
Validation Accuracy = 0.995

EPOCH 1194 ...
Validation Accuracy = 0.995

EPOCH 1195 ...
Validation Accuracy = 0.990

EPOCH 1196 ...
Validation Accuracy = 0.992

EPOCH 1197 ...
Validation Accuracy = 0.994

EPOCH 1198 ...
Validation Accuracy = 0.994

EPOCH 1199 ...
Validation Accuracy = 0.994

EPOCH 1200 ...
Validation Accuracy = 0.995

EPOCH 1201 ...
Validation Accuracy = 0.995

EPOCH 1202 ...
Validation Accuracy = 0.995

EPOCH 1203 ...
Validation Accuracy = 0.995

EPOCH 1204 ...
Validation Accuracy = 0.995

EPOCH 1205 ...
Validation Accuracy = 0.995

EPOCH 1206 ...
Validation Accuracy = 0.995

EPOCH 1207 ...
Validation Accuracy = 0.995

EPOCH 1208 ...
Validation Accuracy = 0.995

EPOCH 1209 ...
Validation Accuracy = 0.995

EPOCH 1210 ...
Validation Accuracy = 0.995

EPOCH 1211 ...
Validation Accuracy = 0.995

EPOCH 1212 ...
Validation Accuracy = 0.995

EPOCH 1213 ...
Validation Accuracy = 0.995

EPOCH 1214 ...
Validation Accuracy = 0.995

EPOCH 1215 ...
Validation Accuracy = 0.995

EPOCH 1216 ...
Validation Accuracy = 0.995

EPOCH 1217 ...
Validation Accuracy = 0.995

EPOCH 1218 ...
Validation Accuracy = 0.995

EPOCH 1219 ...
Validation Accuracy = 0.995

EPOCH 1220 ...
Validation Accuracy = 0.995

EPOCH 1221 ...
Validation Accuracy = 0.995

EPOCH 1222 ...
Validation Accuracy = 0.995

EPOCH 1223 ...
Validation Accuracy = 0.995

EPOCH 1224 ...
Validation Accuracy = 0.995

EPOCH 1225 ...
Validation Accuracy = 0.995

EPOCH 1226 ...
Validation Accuracy = 0.995

EPOCH 1227 ...
Validation Accuracy = 0.995

EPOCH 1228 ...
Validation Accuracy = 0.995

EPOCH 1229 ...
Validation Accuracy = 0.995

EPOCH 1230 ...
Validation Accuracy = 0.995

EPOCH 1231 ...
Validation Accuracy = 0.995

EPOCH 1232 ...
Validation Accuracy = 0.995

EPOCH 1233 ...
Validation Accuracy = 0.995

EPOCH 1234 ...
Validation Accuracy = 0.995

EPOCH 1235 ...
Validation Accuracy = 0.995

EPOCH 1236 ...
Validation Accuracy = 0.995

EPOCH 1237 ...
Validation Accuracy = 0.995

EPOCH 1238 ...
Validation Accuracy = 0.995

EPOCH 1239 ...
Validation Accuracy = 0.995

EPOCH 1240 ...
Validation Accuracy = 0.995

EPOCH 1241 ...
Validation Accuracy = 0.995

EPOCH 1242 ...
Validation Accuracy = 0.995

EPOCH 1243 ...
Validation Accuracy = 0.995

EPOCH 1244 ...
Validation Accuracy = 0.995

EPOCH 1245 ...
Validation Accuracy = 0.995

EPOCH 1246 ...
Validation Accuracy = 0.995

EPOCH 1247 ...
Validation Accuracy = 0.995

EPOCH 1248 ...
Validation Accuracy = 0.995

EPOCH 1249 ...
Validation Accuracy = 0.995

EPOCH 1250 ...
Validation Accuracy = 0.995

EPOCH 1251 ...
Validation Accuracy = 0.995

EPOCH 1252 ...
Validation Accuracy = 0.995

EPOCH 1253 ...
Validation Accuracy = 0.995

EPOCH 1254 ...
Validation Accuracy = 0.995

EPOCH 1255 ...
Validation Accuracy = 0.995

EPOCH 1256 ...
Validation Accuracy = 0.995

EPOCH 1257 ...
Validation Accuracy = 0.995

EPOCH 1258 ...
Validation Accuracy = 0.995

EPOCH 1259 ...
Validation Accuracy = 0.995

EPOCH 1260 ...
Validation Accuracy = 0.995

EPOCH 1261 ...
Validation Accuracy = 0.995

EPOCH 1262 ...
Validation Accuracy = 0.995

EPOCH 1263 ...
Validation Accuracy = 0.995

EPOCH 1264 ...
Validation Accuracy = 0.995

EPOCH 1265 ...
Validation Accuracy = 0.995

EPOCH 1266 ...
Validation Accuracy = 0.995

EPOCH 1267 ...
Validation Accuracy = 0.995

EPOCH 1268 ...
Validation Accuracy = 0.995

EPOCH 1269 ...
Validation Accuracy = 0.995

EPOCH 1270 ...
Validation Accuracy = 0.995

EPOCH 1271 ...
Validation Accuracy = 0.995

EPOCH 1272 ...
Validation Accuracy = 0.995

EPOCH 1273 ...
Validation Accuracy = 0.995

EPOCH 1274 ...
Validation Accuracy = 0.995

EPOCH 1275 ...
Validation Accuracy = 0.995

EPOCH 1276 ...
Validation Accuracy = 0.995

EPOCH 1277 ...
Validation Accuracy = 0.995

EPOCH 1278 ...
Validation Accuracy = 0.995

EPOCH 1279 ...
Validation Accuracy = 0.995

EPOCH 1280 ...
Validation Accuracy = 0.995

EPOCH 1281 ...
Validation Accuracy = 0.995

EPOCH 1282 ...
Validation Accuracy = 0.995

EPOCH 1283 ...
Validation Accuracy = 0.995

EPOCH 1284 ...
Validation Accuracy = 0.995

EPOCH 1285 ...
Validation Accuracy = 0.995

EPOCH 1286 ...
Validation Accuracy = 0.995

EPOCH 1287 ...
Validation Accuracy = 0.995

EPOCH 1288 ...
Validation Accuracy = 0.995

EPOCH 1289 ...
Validation Accuracy = 0.995

EPOCH 1290 ...
Validation Accuracy = 0.995

EPOCH 1291 ...
Validation Accuracy = 0.995

EPOCH 1292 ...
Validation Accuracy = 0.995

EPOCH 1293 ...
Validation Accuracy = 0.995

EPOCH 1294 ...
Validation Accuracy = 0.995

EPOCH 1295 ...
Validation Accuracy = 0.995

EPOCH 1296 ...
Validation Accuracy = 0.995

EPOCH 1297 ...
Validation Accuracy = 0.995

EPOCH 1298 ...
Validation Accuracy = 0.995

EPOCH 1299 ...
Validation Accuracy = 0.995

EPOCH 1300 ...
Validation Accuracy = 0.995

EPOCH 1301 ...
Validation Accuracy = 0.995

EPOCH 1302 ...
Validation Accuracy = 0.995

EPOCH 1303 ...
Validation Accuracy = 0.995

EPOCH 1304 ...
Validation Accuracy = 0.995

EPOCH 1305 ...
Validation Accuracy = 0.995

EPOCH 1306 ...
Validation Accuracy = 0.995

EPOCH 1307 ...
Validation Accuracy = 0.995

EPOCH 1308 ...
Validation Accuracy = 0.990

EPOCH 1309 ...
Validation Accuracy = 0.992

EPOCH 1310 ...
Validation Accuracy = 0.994

EPOCH 1311 ...
Validation Accuracy = 0.995

EPOCH 1312 ...
Validation Accuracy = 0.995

EPOCH 1313 ...
Validation Accuracy = 0.995

EPOCH 1314 ...
Validation Accuracy = 0.995

EPOCH 1315 ...
Validation Accuracy = 0.995

EPOCH 1316 ...
Validation Accuracy = 0.995

EPOCH 1317 ...
Validation Accuracy = 0.995

EPOCH 1318 ...
Validation Accuracy = 0.995

EPOCH 1319 ...
Validation Accuracy = 0.995

EPOCH 1320 ...
Validation Accuracy = 0.995

EPOCH 1321 ...
Validation Accuracy = 0.995

EPOCH 1322 ...
Validation Accuracy = 0.995

EPOCH 1323 ...
Validation Accuracy = 0.995

EPOCH 1324 ...
Validation Accuracy = 0.995

EPOCH 1325 ...
Validation Accuracy = 0.995

EPOCH 1326 ...
Validation Accuracy = 0.995

EPOCH 1327 ...
Validation Accuracy = 0.995

EPOCH 1328 ...
Validation Accuracy = 0.995

EPOCH 1329 ...
Validation Accuracy = 0.995

EPOCH 1330 ...
Validation Accuracy = 0.995

EPOCH 1331 ...
Validation Accuracy = 0.995

EPOCH 1332 ...
Validation Accuracy = 0.995

EPOCH 1333 ...
Validation Accuracy = 0.995

EPOCH 1334 ...
Validation Accuracy = 0.995

EPOCH 1335 ...
Validation Accuracy = 0.995

EPOCH 1336 ...
Validation Accuracy = 0.995

EPOCH 1337 ...
Validation Accuracy = 0.995

EPOCH 1338 ...
Validation Accuracy = 0.995

EPOCH 1339 ...
Validation Accuracy = 0.995

EPOCH 1340 ...
Validation Accuracy = 0.995

EPOCH 1341 ...
Validation Accuracy = 0.995

EPOCH 1342 ...
Validation Accuracy = 0.995

EPOCH 1343 ...
Validation Accuracy = 0.995

EPOCH 1344 ...
Validation Accuracy = 0.995

EPOCH 1345 ...
Validation Accuracy = 0.995

EPOCH 1346 ...
Validation Accuracy = 0.995

EPOCH 1347 ...
Validation Accuracy = 0.995

EPOCH 1348 ...
Validation Accuracy = 0.995

EPOCH 1349 ...
Validation Accuracy = 0.995

EPOCH 1350 ...
Validation Accuracy = 0.995

EPOCH 1351 ...
Validation Accuracy = 0.995

EPOCH 1352 ...
Validation Accuracy = 0.995

EPOCH 1353 ...
Validation Accuracy = 0.995

EPOCH 1354 ...
Validation Accuracy = 0.995

EPOCH 1355 ...
Validation Accuracy = 0.995

EPOCH 1356 ...
Validation Accuracy = 0.995

EPOCH 1357 ...
Validation Accuracy = 0.995

EPOCH 1358 ...
Validation Accuracy = 0.995

EPOCH 1359 ...
Validation Accuracy = 0.995

EPOCH 1360 ...
Validation Accuracy = 0.995

EPOCH 1361 ...
Validation Accuracy = 0.995

EPOCH 1362 ...
Validation Accuracy = 0.995

EPOCH 1363 ...
Validation Accuracy = 0.995

EPOCH 1364 ...
Validation Accuracy = 0.995

EPOCH 1365 ...
Validation Accuracy = 0.995

EPOCH 1366 ...
Validation Accuracy = 0.995

EPOCH 1367 ...
Validation Accuracy = 0.995

EPOCH 1368 ...
Validation Accuracy = 0.995

EPOCH 1369 ...
Validation Accuracy = 0.995

EPOCH 1370 ...
Validation Accuracy = 0.995

EPOCH 1371 ...
Validation Accuracy = 0.995

EPOCH 1372 ...
Validation Accuracy = 0.995

EPOCH 1373 ...
Validation Accuracy = 0.995

EPOCH 1374 ...
Validation Accuracy = 0.995

EPOCH 1375 ...
Validation Accuracy = 0.995

EPOCH 1376 ...
Validation Accuracy = 0.995

EPOCH 1377 ...
Validation Accuracy = 0.995

EPOCH 1378 ...
Validation Accuracy = 0.995

EPOCH 1379 ...
Validation Accuracy = 0.995

EPOCH 1380 ...
Validation Accuracy = 0.995

EPOCH 1381 ...
Validation Accuracy = 0.995

EPOCH 1382 ...
Validation Accuracy = 0.995

EPOCH 1383 ...
Validation Accuracy = 0.995

EPOCH 1384 ...
Validation Accuracy = 0.995

EPOCH 1385 ...
Validation Accuracy = 0.995

EPOCH 1386 ...
Validation Accuracy = 0.995

EPOCH 1387 ...
Validation Accuracy = 0.995

EPOCH 1388 ...
Validation Accuracy = 0.995

EPOCH 1389 ...
Validation Accuracy = 0.995

EPOCH 1390 ...
Validation Accuracy = 0.995

EPOCH 1391 ...
Validation Accuracy = 0.995

EPOCH 1392 ...
Validation Accuracy = 0.995

EPOCH 1393 ...
Validation Accuracy = 0.995

EPOCH 1394 ...
Validation Accuracy = 0.995

EPOCH 1395 ...
Validation Accuracy = 0.995

EPOCH 1396 ...
Validation Accuracy = 0.995

EPOCH 1397 ...
Validation Accuracy = 0.995

EPOCH 1398 ...
Validation Accuracy = 0.995

EPOCH 1399 ...
Validation Accuracy = 0.995

EPOCH 1400 ...
Validation Accuracy = 0.995

EPOCH 1401 ...
Validation Accuracy = 0.995

EPOCH 1402 ...
Validation Accuracy = 0.995

EPOCH 1403 ...
Validation Accuracy = 0.995

EPOCH 1404 ...
Validation Accuracy = 0.995

EPOCH 1405 ...
Validation Accuracy = 0.995

EPOCH 1406 ...
Validation Accuracy = 0.995

EPOCH 1407 ...
Validation Accuracy = 0.995

EPOCH 1408 ...
Validation Accuracy = 0.995

EPOCH 1409 ...
Validation Accuracy = 0.995

EPOCH 1410 ...
Validation Accuracy = 0.995

EPOCH 1411 ...
Validation Accuracy = 0.995

EPOCH 1412 ...
Validation Accuracy = 0.995

EPOCH 1413 ...
Validation Accuracy = 0.995

EPOCH 1414 ...
Validation Accuracy = 0.995

EPOCH 1415 ...
Validation Accuracy = 0.995

EPOCH 1416 ...
Validation Accuracy = 0.995

EPOCH 1417 ...
Validation Accuracy = 0.991

EPOCH 1418 ...
Validation Accuracy = 0.994

EPOCH 1419 ...
Validation Accuracy = 0.995

EPOCH 1420 ...
Validation Accuracy = 0.995

EPOCH 1421 ...
Validation Accuracy = 0.995

EPOCH 1422 ...
Validation Accuracy = 0.995

EPOCH 1423 ...
Validation Accuracy = 0.995

EPOCH 1424 ...
Validation Accuracy = 0.995

EPOCH 1425 ...
Validation Accuracy = 0.995

EPOCH 1426 ...
Validation Accuracy = 0.995

EPOCH 1427 ...
Validation Accuracy = 0.995

EPOCH 1428 ...
Validation Accuracy = 0.995

EPOCH 1429 ...
Validation Accuracy = 0.995

EPOCH 1430 ...
Validation Accuracy = 0.995

EPOCH 1431 ...
Validation Accuracy = 0.995

EPOCH 1432 ...
Validation Accuracy = 0.995

EPOCH 1433 ...
Validation Accuracy = 0.995

EPOCH 1434 ...
Validation Accuracy = 0.995

EPOCH 1435 ...
Validation Accuracy = 0.995

EPOCH 1436 ...
Validation Accuracy = 0.995

EPOCH 1437 ...
Validation Accuracy = 0.995

EPOCH 1438 ...
Validation Accuracy = 0.995

EPOCH 1439 ...
Validation Accuracy = 0.995

EPOCH 1440 ...
Validation Accuracy = 0.995

EPOCH 1441 ...
Validation Accuracy = 0.995

EPOCH 1442 ...
Validation Accuracy = 0.995

EPOCH 1443 ...
Validation Accuracy = 0.995

EPOCH 1444 ...
Validation Accuracy = 0.995

EPOCH 1445 ...
Validation Accuracy = 0.995

EPOCH 1446 ...
Validation Accuracy = 0.995

EPOCH 1447 ...
Validation Accuracy = 0.995

EPOCH 1448 ...
Validation Accuracy = 0.995

EPOCH 1449 ...
Validation Accuracy = 0.995

EPOCH 1450 ...
Validation Accuracy = 0.995

EPOCH 1451 ...
Validation Accuracy = 0.995

EPOCH 1452 ...
Validation Accuracy = 0.995

EPOCH 1453 ...
Validation Accuracy = 0.995

EPOCH 1454 ...
Validation Accuracy = 0.995

EPOCH 1455 ...
Validation Accuracy = 0.995

EPOCH 1456 ...
Validation Accuracy = 0.995

EPOCH 1457 ...
Validation Accuracy = 0.995

EPOCH 1458 ...
Validation Accuracy = 0.995

EPOCH 1459 ...
Validation Accuracy = 0.995

EPOCH 1460 ...
Validation Accuracy = 0.995

EPOCH 1461 ...
Validation Accuracy = 0.995

EPOCH 1462 ...
Validation Accuracy = 0.995

EPOCH 1463 ...
Validation Accuracy = 0.995

EPOCH 1464 ...
Validation Accuracy = 0.995

EPOCH 1465 ...
Validation Accuracy = 0.995

EPOCH 1466 ...
Validation Accuracy = 0.995

EPOCH 1467 ...
Validation Accuracy = 0.995

EPOCH 1468 ...
Validation Accuracy = 0.995

EPOCH 1469 ...
Validation Accuracy = 0.995

EPOCH 1470 ...
Validation Accuracy = 0.995

EPOCH 1471 ...
Validation Accuracy = 0.995

EPOCH 1472 ...
Validation Accuracy = 0.995

EPOCH 1473 ...
Validation Accuracy = 0.995

EPOCH 1474 ...
Validation Accuracy = 0.995

EPOCH 1475 ...
Validation Accuracy = 0.995

EPOCH 1476 ...
Validation Accuracy = 0.995

EPOCH 1477 ...
Validation Accuracy = 0.995

EPOCH 1478 ...
Validation Accuracy = 0.995

EPOCH 1479 ...
Validation Accuracy = 0.995

EPOCH 1480 ...
Validation Accuracy = 0.995

EPOCH 1481 ...
Validation Accuracy = 0.995

EPOCH 1482 ...
Validation Accuracy = 0.995

EPOCH 1483 ...
Validation Accuracy = 0.995

EPOCH 1484 ...
Validation Accuracy = 0.995

EPOCH 1485 ...
Validation Accuracy = 0.995

EPOCH 1486 ...
Validation Accuracy = 0.995

EPOCH 1487 ...
Validation Accuracy = 0.995

EPOCH 1488 ...
Validation Accuracy = 0.995

EPOCH 1489 ...
Validation Accuracy = 0.995

EPOCH 1490 ...
Validation Accuracy = 0.995

EPOCH 1491 ...
Validation Accuracy = 0.995

EPOCH 1492 ...
Validation Accuracy = 0.995

EPOCH 1493 ...
Validation Accuracy = 0.995

EPOCH 1494 ...
Validation Accuracy = 0.995

EPOCH 1495 ...
Validation Accuracy = 0.995

EPOCH 1496 ...
Validation Accuracy = 0.995

EPOCH 1497 ...
Validation Accuracy = 0.995

EPOCH 1498 ...
Validation Accuracy = 0.995

EPOCH 1499 ...
Validation Accuracy = 0.995

EPOCH 1500 ...
Validation Accuracy = 0.995

EPOCH 1501 ...
Validation Accuracy = 0.995

EPOCH 1502 ...
Validation Accuracy = 0.995

EPOCH 1503 ...
Validation Accuracy = 0.995

EPOCH 1504 ...
Validation Accuracy = 0.995

EPOCH 1505 ...
Validation Accuracy = 0.995

EPOCH 1506 ...
Validation Accuracy = 0.995

EPOCH 1507 ...
Validation Accuracy = 0.995

EPOCH 1508 ...
Validation Accuracy = 0.995

EPOCH 1509 ...
Validation Accuracy = 0.995

EPOCH 1510 ...
Validation Accuracy = 0.995

EPOCH 1511 ...
Validation Accuracy = 0.995

EPOCH 1512 ...
Validation Accuracy = 0.995

EPOCH 1513 ...
Validation Accuracy = 0.995

EPOCH 1514 ...
Validation Accuracy = 0.995

EPOCH 1515 ...
Validation Accuracy = 0.995

EPOCH 1516 ...
Validation Accuracy = 0.995

EPOCH 1517 ...
Validation Accuracy = 0.995

EPOCH 1518 ...
Validation Accuracy = 0.995

EPOCH 1519 ...
Validation Accuracy = 0.995

EPOCH 1520 ...
Validation Accuracy = 0.995

EPOCH 1521 ...
Validation Accuracy = 0.995

EPOCH 1522 ...
Validation Accuracy = 0.995

EPOCH 1523 ...
Validation Accuracy = 0.995

EPOCH 1524 ...
Validation Accuracy = 0.995

EPOCH 1525 ...
Validation Accuracy = 0.995

EPOCH 1526 ...
Validation Accuracy = 0.995

EPOCH 1527 ...
Validation Accuracy = 0.995

EPOCH 1528 ...
Validation Accuracy = 0.995

EPOCH 1529 ...
Validation Accuracy = 0.995

EPOCH 1530 ...
Validation Accuracy = 0.995

EPOCH 1531 ...
Validation Accuracy = 0.995

EPOCH 1532 ...
Validation Accuracy = 0.995

EPOCH 1533 ...
Validation Accuracy = 0.995

EPOCH 1534 ...
Validation Accuracy = 0.995

EPOCH 1535 ...
Validation Accuracy = 0.990

EPOCH 1536 ...
Validation Accuracy = 0.993

EPOCH 1537 ...
Validation Accuracy = 0.994

EPOCH 1538 ...
Validation Accuracy = 0.994

EPOCH 1539 ...
Validation Accuracy = 0.995

EPOCH 1540 ...
Validation Accuracy = 0.995

EPOCH 1541 ...
Validation Accuracy = 0.995

EPOCH 1542 ...
Validation Accuracy = 0.995

EPOCH 1543 ...
Validation Accuracy = 0.995

EPOCH 1544 ...
Validation Accuracy = 0.995

EPOCH 1545 ...
Validation Accuracy = 0.995

EPOCH 1546 ...
Validation Accuracy = 0.995

EPOCH 1547 ...
Validation Accuracy = 0.995

EPOCH 1548 ...
Validation Accuracy = 0.995

EPOCH 1549 ...
Validation Accuracy = 0.995

EPOCH 1550 ...
Validation Accuracy = 0.995

EPOCH 1551 ...
Validation Accuracy = 0.995

EPOCH 1552 ...
Validation Accuracy = 0.995

EPOCH 1553 ...
Validation Accuracy = 0.995

EPOCH 1554 ...
Validation Accuracy = 0.995

EPOCH 1555 ...
Validation Accuracy = 0.995

EPOCH 1556 ...
Validation Accuracy = 0.995

EPOCH 1557 ...
Validation Accuracy = 0.995

EPOCH 1558 ...
Validation Accuracy = 0.995

EPOCH 1559 ...
Validation Accuracy = 0.995

EPOCH 1560 ...
Validation Accuracy = 0.995

EPOCH 1561 ...
Validation Accuracy = 0.995

EPOCH 1562 ...
Validation Accuracy = 0.995

EPOCH 1563 ...
Validation Accuracy = 0.995

EPOCH 1564 ...
Validation Accuracy = 0.995

EPOCH 1565 ...
Validation Accuracy = 0.995

EPOCH 1566 ...
Validation Accuracy = 0.995

EPOCH 1567 ...
Validation Accuracy = 0.995

EPOCH 1568 ...
Validation Accuracy = 0.995

EPOCH 1569 ...
Validation Accuracy = 0.995

EPOCH 1570 ...
Validation Accuracy = 0.995

EPOCH 1571 ...
Validation Accuracy = 0.995

EPOCH 1572 ...
Validation Accuracy = 0.995

EPOCH 1573 ...
Validation Accuracy = 0.995

EPOCH 1574 ...
Validation Accuracy = 0.995

EPOCH 1575 ...
Validation Accuracy = 0.995

EPOCH 1576 ...
Validation Accuracy = 0.995

EPOCH 1577 ...
Validation Accuracy = 0.995

EPOCH 1578 ...
Validation Accuracy = 0.995

EPOCH 1579 ...
Validation Accuracy = 0.995

EPOCH 1580 ...
Validation Accuracy = 0.995

EPOCH 1581 ...
Validation Accuracy = 0.995

EPOCH 1582 ...
Validation Accuracy = 0.995

EPOCH 1583 ...
Validation Accuracy = 0.995

EPOCH 1584 ...
Validation Accuracy = 0.995

EPOCH 1585 ...
Validation Accuracy = 0.995

EPOCH 1586 ...
Validation Accuracy = 0.995

EPOCH 1587 ...
Validation Accuracy = 0.995

EPOCH 1588 ...
Validation Accuracy = 0.995

EPOCH 1589 ...
Validation Accuracy = 0.995

EPOCH 1590 ...
Validation Accuracy = 0.995

EPOCH 1591 ...
Validation Accuracy = 0.995

EPOCH 1592 ...
Validation Accuracy = 0.995

EPOCH 1593 ...
Validation Accuracy = 0.995

EPOCH 1594 ...
Validation Accuracy = 0.995

EPOCH 1595 ...
Validation Accuracy = 0.995

EPOCH 1596 ...
Validation Accuracy = 0.995

EPOCH 1597 ...
Validation Accuracy = 0.995

EPOCH 1598 ...
Validation Accuracy = 0.995

EPOCH 1599 ...
Validation Accuracy = 0.995

EPOCH 1600 ...
Validation Accuracy = 0.995

EPOCH 1601 ...
Validation Accuracy = 0.995

EPOCH 1602 ...
Validation Accuracy = 0.995

EPOCH 1603 ...
Validation Accuracy = 0.995

EPOCH 1604 ...
Validation Accuracy = 0.995

EPOCH 1605 ...
Validation Accuracy = 0.995

EPOCH 1606 ...
Validation Accuracy = 0.995

EPOCH 1607 ...
Validation Accuracy = 0.995

EPOCH 1608 ...
Validation Accuracy = 0.996

EPOCH 1609 ...
Validation Accuracy = 0.995

EPOCH 1610 ...
Validation Accuracy = 0.995

EPOCH 1611 ...
Validation Accuracy = 0.995

EPOCH 1612 ...
Validation Accuracy = 0.996

EPOCH 1613 ...
Validation Accuracy = 0.996

EPOCH 1614 ...
Validation Accuracy = 0.996

EPOCH 1615 ...
Validation Accuracy = 0.996

EPOCH 1616 ...
Validation Accuracy = 0.996

EPOCH 1617 ...
Validation Accuracy = 0.995

EPOCH 1618 ...
Validation Accuracy = 0.996

EPOCH 1619 ...
Validation Accuracy = 0.996

EPOCH 1620 ...
Validation Accuracy = 0.996

EPOCH 1621 ...
Validation Accuracy = 0.996

EPOCH 1622 ...
Validation Accuracy = 0.996

EPOCH 1623 ...
Validation Accuracy = 0.996

EPOCH 1624 ...
Validation Accuracy = 0.996

EPOCH 1625 ...
Validation Accuracy = 0.996

EPOCH 1626 ...
Validation Accuracy = 0.995

EPOCH 1627 ...
Validation Accuracy = 0.996

EPOCH 1628 ...
Validation Accuracy = 0.996

EPOCH 1629 ...
Validation Accuracy = 0.996

EPOCH 1630 ...
Validation Accuracy = 0.996

EPOCH 1631 ...
Validation Accuracy = 0.995

EPOCH 1632 ...
Validation Accuracy = 0.995

EPOCH 1633 ...
Validation Accuracy = 0.995

EPOCH 1634 ...
Validation Accuracy = 0.995

EPOCH 1635 ...
Validation Accuracy = 0.995

EPOCH 1636 ...
Validation Accuracy = 0.995

EPOCH 1637 ...
Validation Accuracy = 0.996

EPOCH 1638 ...
Validation Accuracy = 0.995

EPOCH 1639 ...
Validation Accuracy = 0.995

EPOCH 1640 ...
Validation Accuracy = 0.995

EPOCH 1641 ...
Validation Accuracy = 0.995

EPOCH 1642 ...
Validation Accuracy = 0.995

EPOCH 1643 ...
Validation Accuracy = 0.996

EPOCH 1644 ...
Validation Accuracy = 0.995

EPOCH 1645 ...
Validation Accuracy = 0.995

EPOCH 1646 ...
Validation Accuracy = 0.995

EPOCH 1647 ...
Validation Accuracy = 0.995

EPOCH 1648 ...
Validation Accuracy = 0.995

EPOCH 1649 ...
Validation Accuracy = 0.995

EPOCH 1650 ...
Validation Accuracy = 0.995

EPOCH 1651 ...
Validation Accuracy = 0.995

EPOCH 1652 ...
Validation Accuracy = 0.995

EPOCH 1653 ...
Validation Accuracy = 0.995

EPOCH 1654 ...
Validation Accuracy = 0.995

EPOCH 1655 ...
Validation Accuracy = 0.995

EPOCH 1656 ...
Validation Accuracy = 0.995

EPOCH 1657 ...
Validation Accuracy = 0.995

EPOCH 1658 ...
Validation Accuracy = 0.995

EPOCH 1659 ...
Validation Accuracy = 0.995

EPOCH 1660 ...
Validation Accuracy = 0.995

EPOCH 1661 ...
Validation Accuracy = 0.989

EPOCH 1662 ...
Validation Accuracy = 0.993

EPOCH 1663 ...
Validation Accuracy = 0.995

EPOCH 1664 ...
Validation Accuracy = 0.995

EPOCH 1665 ...
Validation Accuracy = 0.995

EPOCH 1666 ...
Validation Accuracy = 0.995

EPOCH 1667 ...
Validation Accuracy = 0.995

EPOCH 1668 ...
Validation Accuracy = 0.995

EPOCH 1669 ...
Validation Accuracy = 0.995

EPOCH 1670 ...
Validation Accuracy = 0.995

EPOCH 1671 ...
Validation Accuracy = 0.995

EPOCH 1672 ...
Validation Accuracy = 0.995

EPOCH 1673 ...
Validation Accuracy = 0.995

EPOCH 1674 ...
Validation Accuracy = 0.995

EPOCH 1675 ...
Validation Accuracy = 0.995

EPOCH 1676 ...
Validation Accuracy = 0.995

EPOCH 1677 ...
Validation Accuracy = 0.995

EPOCH 1678 ...
Validation Accuracy = 0.995

EPOCH 1679 ...
Validation Accuracy = 0.995

EPOCH 1680 ...
Validation Accuracy = 0.995

EPOCH 1681 ...
Validation Accuracy = 0.995

EPOCH 1682 ...
Validation Accuracy = 0.995

EPOCH 1683 ...
Validation Accuracy = 0.995

EPOCH 1684 ...
Validation Accuracy = 0.995

EPOCH 1685 ...
Validation Accuracy = 0.995

EPOCH 1686 ...
Validation Accuracy = 0.995

EPOCH 1687 ...
Validation Accuracy = 0.995

EPOCH 1688 ...
Validation Accuracy = 0.995

EPOCH 1689 ...
Validation Accuracy = 0.995

EPOCH 1690 ...
Validation Accuracy = 0.995

EPOCH 1691 ...
Validation Accuracy = 0.995

EPOCH 1692 ...
Validation Accuracy = 0.995

EPOCH 1693 ...
Validation Accuracy = 0.995

EPOCH 1694 ...
Validation Accuracy = 0.995

EPOCH 1695 ...
Validation Accuracy = 0.995

EPOCH 1696 ...
Validation Accuracy = 0.995

EPOCH 1697 ...
Validation Accuracy = 0.995

EPOCH 1698 ...
Validation Accuracy = 0.995

EPOCH 1699 ...
Validation Accuracy = 0.995

EPOCH 1700 ...
Validation Accuracy = 0.995

EPOCH 1701 ...
Validation Accuracy = 0.995

EPOCH 1702 ...
Validation Accuracy = 0.995

EPOCH 1703 ...
Validation Accuracy = 0.995

EPOCH 1704 ...
Validation Accuracy = 0.995

EPOCH 1705 ...
Validation Accuracy = 0.995

EPOCH 1706 ...
Validation Accuracy = 0.995

EPOCH 1707 ...
Validation Accuracy = 0.995

EPOCH 1708 ...
Validation Accuracy = 0.995

EPOCH 1709 ...
Validation Accuracy = 0.995

EPOCH 1710 ...
Validation Accuracy = 0.995

EPOCH 1711 ...
Validation Accuracy = 0.995

EPOCH 1712 ...
Validation Accuracy = 0.995

EPOCH 1713 ...
Validation Accuracy = 0.995

EPOCH 1714 ...
Validation Accuracy = 0.995

EPOCH 1715 ...
Validation Accuracy = 0.995

EPOCH 1716 ...
Validation Accuracy = 0.995

EPOCH 1717 ...
Validation Accuracy = 0.995

EPOCH 1718 ...
Validation Accuracy = 0.995

EPOCH 1719 ...
Validation Accuracy = 0.995

EPOCH 1720 ...
Validation Accuracy = 0.995

EPOCH 1721 ...
Validation Accuracy = 0.995

EPOCH 1722 ...
Validation Accuracy = 0.995

EPOCH 1723 ...
Validation Accuracy = 0.995

EPOCH 1724 ...
Validation Accuracy = 0.995

EPOCH 1725 ...
Validation Accuracy = 0.995

EPOCH 1726 ...
Validation Accuracy = 0.995

EPOCH 1727 ...
Validation Accuracy = 0.995

EPOCH 1728 ...
Validation Accuracy = 0.995

EPOCH 1729 ...
Validation Accuracy = 0.995

EPOCH 1730 ...
Validation Accuracy = 0.995

EPOCH 1731 ...
Validation Accuracy = 0.995

EPOCH 1732 ...
Validation Accuracy = 0.995

EPOCH 1733 ...
Validation Accuracy = 0.995

EPOCH 1734 ...
Validation Accuracy = 0.995

EPOCH 1735 ...
Validation Accuracy = 0.995

EPOCH 1736 ...
Validation Accuracy = 0.995

EPOCH 1737 ...
Validation Accuracy = 0.995

EPOCH 1738 ...
Validation Accuracy = 0.995

EPOCH 1739 ...
Validation Accuracy = 0.995

EPOCH 1740 ...
Validation Accuracy = 0.995

EPOCH 1741 ...
Validation Accuracy = 0.995

EPOCH 1742 ...
Validation Accuracy = 0.995

EPOCH 1743 ...
Validation Accuracy = 0.995

EPOCH 1744 ...
Validation Accuracy = 0.995

EPOCH 1745 ...
Validation Accuracy = 0.995

EPOCH 1746 ...
Validation Accuracy = 0.995

EPOCH 1747 ...
Validation Accuracy = 0.995

EPOCH 1748 ...
Validation Accuracy = 0.995

EPOCH 1749 ...
Validation Accuracy = 0.995

EPOCH 1750 ...
Validation Accuracy = 0.995

EPOCH 1751 ...
Validation Accuracy = 0.995

EPOCH 1752 ...
Validation Accuracy = 0.995

EPOCH 1753 ...
Validation Accuracy = 0.995

EPOCH 1754 ...
Validation Accuracy = 0.995

EPOCH 1755 ...
Validation Accuracy = 0.995

EPOCH 1756 ...
Validation Accuracy = 0.995

EPOCH 1757 ...
Validation Accuracy = 0.995

EPOCH 1758 ...
Validation Accuracy = 0.995

EPOCH 1759 ...
Validation Accuracy = 0.995

EPOCH 1760 ...
Validation Accuracy = 0.995

EPOCH 1761 ...
Validation Accuracy = 0.995

EPOCH 1762 ...
Validation Accuracy = 0.995

EPOCH 1763 ...
Validation Accuracy = 0.995

EPOCH 1764 ...
Validation Accuracy = 0.995

EPOCH 1765 ...
Validation Accuracy = 0.995

EPOCH 1766 ...
Validation Accuracy = 0.995

EPOCH 1767 ...
Validation Accuracy = 0.995

EPOCH 1768 ...
Validation Accuracy = 0.995

EPOCH 1769 ...
Validation Accuracy = 0.995

EPOCH 1770 ...
Validation Accuracy = 0.995

EPOCH 1771 ...
Validation Accuracy = 0.995

EPOCH 1772 ...
Validation Accuracy = 0.995

EPOCH 1773 ...
Validation Accuracy = 0.995

EPOCH 1774 ...
Validation Accuracy = 0.995

EPOCH 1775 ...
Validation Accuracy = 0.995

EPOCH 1776 ...
Validation Accuracy = 0.995

EPOCH 1777 ...
Validation Accuracy = 0.995

EPOCH 1778 ...
Validation Accuracy = 0.995

EPOCH 1779 ...
Validation Accuracy = 0.995

EPOCH 1780 ...
Validation Accuracy = 0.995

EPOCH 1781 ...
Validation Accuracy = 0.995

EPOCH 1782 ...
Validation Accuracy = 0.995

EPOCH 1783 ...
Validation Accuracy = 0.995

EPOCH 1784 ...
Validation Accuracy = 0.995

EPOCH 1785 ...
Validation Accuracy = 0.995

EPOCH 1786 ...
Validation Accuracy = 0.995

EPOCH 1787 ...
Validation Accuracy = 0.995

EPOCH 1788 ...
Validation Accuracy = 0.995

EPOCH 1789 ...
Validation Accuracy = 0.995

EPOCH 1790 ...
Validation Accuracy = 0.996

EPOCH 1791 ...
Validation Accuracy = 0.995

EPOCH 1792 ...
Validation Accuracy = 0.995

EPOCH 1793 ...
Validation Accuracy = 0.995

EPOCH 1794 ...
Validation Accuracy = 0.995

EPOCH 1795 ...
Validation Accuracy = 0.984

EPOCH 1796 ...
Validation Accuracy = 0.991

EPOCH 1797 ...
Validation Accuracy = 0.994

EPOCH 1798 ...
Validation Accuracy = 0.994

EPOCH 1799 ...
Validation Accuracy = 0.995

EPOCH 1800 ...
Validation Accuracy = 0.995

EPOCH 1801 ...
Validation Accuracy = 0.995

EPOCH 1802 ...
Validation Accuracy = 0.995

EPOCH 1803 ...
Validation Accuracy = 0.995

EPOCH 1804 ...
Validation Accuracy = 0.995

EPOCH 1805 ...
Validation Accuracy = 0.995

EPOCH 1806 ...
Validation Accuracy = 0.995

EPOCH 1807 ...
Validation Accuracy = 0.995

EPOCH 1808 ...
Validation Accuracy = 0.995

EPOCH 1809 ...
Validation Accuracy = 0.995

EPOCH 1810 ...
Validation Accuracy = 0.995

EPOCH 1811 ...
Validation Accuracy = 0.995

EPOCH 1812 ...
Validation Accuracy = 0.995

EPOCH 1813 ...
Validation Accuracy = 0.995

EPOCH 1814 ...
Validation Accuracy = 0.995

EPOCH 1815 ...
Validation Accuracy = 0.995

EPOCH 1816 ...
Validation Accuracy = 0.995

EPOCH 1817 ...
Validation Accuracy = 0.995

EPOCH 1818 ...
Validation Accuracy = 0.995

EPOCH 1819 ...
Validation Accuracy = 0.995

EPOCH 1820 ...
Validation Accuracy = 0.995

EPOCH 1821 ...
Validation Accuracy = 0.995

EPOCH 1822 ...
Validation Accuracy = 0.995

EPOCH 1823 ...
Validation Accuracy = 0.995

EPOCH 1824 ...
Validation Accuracy = 0.995

EPOCH 1825 ...
Validation Accuracy = 0.995

EPOCH 1826 ...
Validation Accuracy = 0.995

EPOCH 1827 ...
Validation Accuracy = 0.995

EPOCH 1828 ...
Validation Accuracy = 0.995

EPOCH 1829 ...
Validation Accuracy = 0.995

EPOCH 1830 ...
Validation Accuracy = 0.995

EPOCH 1831 ...
Validation Accuracy = 0.995

EPOCH 1832 ...
Validation Accuracy = 0.995

EPOCH 1833 ...
Validation Accuracy = 0.995

EPOCH 1834 ...
Validation Accuracy = 0.995

EPOCH 1835 ...
Validation Accuracy = 0.995

EPOCH 1836 ...
Validation Accuracy = 0.995

EPOCH 1837 ...
Validation Accuracy = 0.995

EPOCH 1838 ...
Validation Accuracy = 0.995

EPOCH 1839 ...
Validation Accuracy = 0.995

EPOCH 1840 ...
Validation Accuracy = 0.995

EPOCH 1841 ...
Validation Accuracy = 0.995

EPOCH 1842 ...
Validation Accuracy = 0.995

EPOCH 1843 ...
Validation Accuracy = 0.995

EPOCH 1844 ...
Validation Accuracy = 0.995

EPOCH 1845 ...
Validation Accuracy = 0.995

EPOCH 1846 ...
Validation Accuracy = 0.995

EPOCH 1847 ...
Validation Accuracy = 0.995

EPOCH 1848 ...
Validation Accuracy = 0.995

EPOCH 1849 ...
Validation Accuracy = 0.995

EPOCH 1850 ...
Validation Accuracy = 0.995

EPOCH 1851 ...
Validation Accuracy = 0.995

EPOCH 1852 ...
Validation Accuracy = 0.995

EPOCH 1853 ...
Validation Accuracy = 0.995

EPOCH 1854 ...
Validation Accuracy = 0.995

EPOCH 1855 ...
Validation Accuracy = 0.995

EPOCH 1856 ...
Validation Accuracy = 0.995

EPOCH 1857 ...
Validation Accuracy = 0.995

EPOCH 1858 ...
Validation Accuracy = 0.995

EPOCH 1859 ...
Validation Accuracy = 0.995

EPOCH 1860 ...
Validation Accuracy = 0.995

EPOCH 1861 ...
Validation Accuracy = 0.995

EPOCH 1862 ...
Validation Accuracy = 0.995

EPOCH 1863 ...
Validation Accuracy = 0.996

EPOCH 1864 ...
Validation Accuracy = 0.996

EPOCH 1865 ...
Validation Accuracy = 0.996

EPOCH 1866 ...
Validation Accuracy = 0.996

EPOCH 1867 ...
Validation Accuracy = 0.996

EPOCH 1868 ...
Validation Accuracy = 0.996

EPOCH 1869 ...
Validation Accuracy = 0.996

EPOCH 1870 ...
Validation Accuracy = 0.996

EPOCH 1871 ...
Validation Accuracy = 0.996

EPOCH 1872 ...
Validation Accuracy = 0.996

EPOCH 1873 ...
Validation Accuracy = 0.996

EPOCH 1874 ...
Validation Accuracy = 0.996

EPOCH 1875 ...
Validation Accuracy = 0.996

EPOCH 1876 ...
Validation Accuracy = 0.996

EPOCH 1877 ...
Validation Accuracy = 0.996

EPOCH 1878 ...
Validation Accuracy = 0.996

EPOCH 1879 ...
Validation Accuracy = 0.996

EPOCH 1880 ...
Validation Accuracy = 0.996

EPOCH 1881 ...
Validation Accuracy = 0.996

EPOCH 1882 ...
Validation Accuracy = 0.996

EPOCH 1883 ...
Validation Accuracy = 0.996

EPOCH 1884 ...
Validation Accuracy = 0.996

EPOCH 1885 ...
Validation Accuracy = 0.996

EPOCH 1886 ...
Validation Accuracy = 0.996

EPOCH 1887 ...
Validation Accuracy = 0.996

EPOCH 1888 ...
Validation Accuracy = 0.996

EPOCH 1889 ...
Validation Accuracy = 0.996

EPOCH 1890 ...
Validation Accuracy = 0.996

EPOCH 1891 ...
Validation Accuracy = 0.996

EPOCH 1892 ...
Validation Accuracy = 0.996

EPOCH 1893 ...
Validation Accuracy = 0.996

EPOCH 1894 ...
Validation Accuracy = 0.996

EPOCH 1895 ...
Validation Accuracy = 0.996

EPOCH 1896 ...
Validation Accuracy = 0.996

EPOCH 1897 ...
Validation Accuracy = 0.996

EPOCH 1898 ...
Validation Accuracy = 0.996

EPOCH 1899 ...
Validation Accuracy = 0.996

EPOCH 1900 ...
Validation Accuracy = 0.996

EPOCH 1901 ...
Validation Accuracy = 0.996

EPOCH 1902 ...
Validation Accuracy = 0.996

EPOCH 1903 ...
Validation Accuracy = 0.996

EPOCH 1904 ...
Validation Accuracy = 0.995

EPOCH 1905 ...
Validation Accuracy = 0.996

EPOCH 1906 ...
Validation Accuracy = 0.995

EPOCH 1907 ...
Validation Accuracy = 0.996

EPOCH 1908 ...
Validation Accuracy = 0.995

EPOCH 1909 ...
Validation Accuracy = 0.995

EPOCH 1910 ...
Validation Accuracy = 0.995

EPOCH 1911 ...
Validation Accuracy = 0.995

EPOCH 1912 ...
Validation Accuracy = 0.996

EPOCH 1913 ...
Validation Accuracy = 0.995

EPOCH 1914 ...
Validation Accuracy = 0.995

EPOCH 1915 ...
Validation Accuracy = 0.996

EPOCH 1916 ...
Validation Accuracy = 0.996

EPOCH 1917 ...
Validation Accuracy = 0.996

EPOCH 1918 ...
Validation Accuracy = 0.995

EPOCH 1919 ...
Validation Accuracy = 0.995

EPOCH 1920 ...
Validation Accuracy = 0.996

EPOCH 1921 ...
Validation Accuracy = 0.996

EPOCH 1922 ...
Validation Accuracy = 0.996

EPOCH 1923 ...
Validation Accuracy = 0.995

EPOCH 1924 ...
Validation Accuracy = 0.995

EPOCH 1925 ...
Validation Accuracy = 0.995

EPOCH 1926 ...
Validation Accuracy = 0.995

EPOCH 1927 ...
Validation Accuracy = 0.996

EPOCH 1928 ...
Validation Accuracy = 0.995

EPOCH 1929 ...
Validation Accuracy = 0.996

EPOCH 1930 ...
Validation Accuracy = 0.995

EPOCH 1931 ...
Validation Accuracy = 0.995

EPOCH 1932 ...
Validation Accuracy = 0.996

EPOCH 1933 ...
Validation Accuracy = 0.995

EPOCH 1934 ...
Validation Accuracy = 0.995

EPOCH 1935 ...
Validation Accuracy = 0.996

EPOCH 1936 ...
Validation Accuracy = 0.996

EPOCH 1937 ...
Validation Accuracy = 0.996

EPOCH 1938 ...
Validation Accuracy = 0.996

EPOCH 1939 ...
Validation Accuracy = 0.995

EPOCH 1940 ...
Validation Accuracy = 0.995

EPOCH 1941 ...
Validation Accuracy = 0.995

EPOCH 1942 ...
Validation Accuracy = 0.995

EPOCH 1943 ...
Validation Accuracy = 0.996

EPOCH 1944 ...
Validation Accuracy = 0.995

EPOCH 1945 ...
Validation Accuracy = 0.995

EPOCH 1946 ...
Validation Accuracy = 0.996

EPOCH 1947 ...
Validation Accuracy = 0.995

EPOCH 1948 ...
Validation Accuracy = 0.995

EPOCH 1949 ...
Validation Accuracy = 0.991

EPOCH 1950 ...
Validation Accuracy = 0.994

EPOCH 1951 ...
Validation Accuracy = 0.995

EPOCH 1952 ...
Validation Accuracy = 0.995

EPOCH 1953 ...
Validation Accuracy = 0.995

EPOCH 1954 ...
Validation Accuracy = 0.995

EPOCH 1955 ...
Validation Accuracy = 0.995

EPOCH 1956 ...
Validation Accuracy = 0.995

EPOCH 1957 ...
Validation Accuracy = 0.995

EPOCH 1958 ...
Validation Accuracy = 0.995

EPOCH 1959 ...
Validation Accuracy = 0.995

EPOCH 1960 ...
Validation Accuracy = 0.995

EPOCH 1961 ...
Validation Accuracy = 0.995

EPOCH 1962 ...
Validation Accuracy = 0.995

EPOCH 1963 ...
Validation Accuracy = 0.995

EPOCH 1964 ...
Validation Accuracy = 0.995

EPOCH 1965 ...
Validation Accuracy = 0.995

EPOCH 1966 ...
Validation Accuracy = 0.995

EPOCH 1967 ...
Validation Accuracy = 0.995

EPOCH 1968 ...
Validation Accuracy = 0.995

EPOCH 1969 ...
Validation Accuracy = 0.995

EPOCH 1970 ...
Validation Accuracy = 0.995

EPOCH 1971 ...
Validation Accuracy = 0.995

EPOCH 1972 ...
Validation Accuracy = 0.995

EPOCH 1973 ...
Validation Accuracy = 0.995

EPOCH 1974 ...
Validation Accuracy = 0.995

EPOCH 1975 ...
Validation Accuracy = 0.995

EPOCH 1976 ...
Validation Accuracy = 0.995

EPOCH 1977 ...
Validation Accuracy = 0.995

EPOCH 1978 ...
Validation Accuracy = 0.995

EPOCH 1979 ...
Validation Accuracy = 0.995

EPOCH 1980 ...
Validation Accuracy = 0.995

EPOCH 1981 ...
Validation Accuracy = 0.995

EPOCH 1982 ...
Validation Accuracy = 0.995

EPOCH 1983 ...
Validation Accuracy = 0.995

EPOCH 1984 ...
Validation Accuracy = 0.995

EPOCH 1985 ...
Validation Accuracy = 0.995

EPOCH 1986 ...
Validation Accuracy = 0.995

EPOCH 1987 ...
Validation Accuracy = 0.995

EPOCH 1988 ...
Validation Accuracy = 0.995

EPOCH 1989 ...
Validation Accuracy = 0.995

EPOCH 1990 ...
Validation Accuracy = 0.995

EPOCH 1991 ...
Validation Accuracy = 0.995

EPOCH 1992 ...
Validation Accuracy = 0.995

EPOCH 1993 ...
Validation Accuracy = 0.995

EPOCH 1994 ...
Validation Accuracy = 0.995

EPOCH 1995 ...
Validation Accuracy = 0.995

EPOCH 1996 ...
Validation Accuracy = 0.995

EPOCH 1997 ...
Validation Accuracy = 0.995

EPOCH 1998 ...
Validation Accuracy = 0.995

EPOCH 1999 ...
Validation Accuracy = 0.995

EPOCH 2000 ...
Validation Accuracy = 0.995

Model saved
In [14]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    test_accuracy = evaluate(X_test_normalized, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))
Test Accuracy = 0.900

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

This is the part of this homework that took the longest. I've spent several days tweaking it and trying to obtain the best results I could.

I tried using the tf.train.GradientDescentOptimizer optimizer with learning rates around [0.1,0.01] and saw that the training set wasn't really learning well. I'm yet to figure out why, as I dropped it in favor of the tf.train.AdamOptimizer optimizer with learning rates around [0.001,0.0001] which yielded such good results I just ran with it.

I was able to make the batch size grow to 2048 for better performance, didn't try anything above this as this was satisfactory in the hardware I used.

I noticed that with the epochs, there was a point after which my model wasn't learning much more, that was around 750 iterations. I tried running up to 2000 iterations in order to realize this.

Another parameter I used was keep_probability given that I'm using dropout operations in my LeNet model. For training I set keep probability to 0.5, and for validation and testing I use 1.0.

Question 5

What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.

Answer:

It was definitely trial-and-error. I did use the LeNet code I had from a previous assignment, and tried expanding it with the Sermanet-Yann paper information I had, but that was very painful as my lack of experience with tensorflow, python, and neural networks were in my way. The first attempts at running this didn't yield the results I was hoping for, with a test accuracy of 50%. After upgrading the LeNet based of the whitepaper, and realizing that I needed to normalize the data, I found that accuracy went up to 80%. After that, I also removed the code I had that was cropping the images based on the train['coords'] and train['sizes'] data and got the test to score 90% accuracy.

I'm not sure why cropping the data didn't yield the results I expected, I plan on investigating this on my spare time. It hurt my test score by a whole 10 percentage points!

After that, I only played with changing the optimizer algorithm, the learning rate, the batch size and the amount of epochs and I got to satisfactory levels. Not perfect, but satisfactory.


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [19]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
import matplotlib.image as mpimg

print("Left: web image, Center: web image scaled to 32x32, Right: example image from original data set in the same class")

web_images = []
web_classes = [23,40,14,28,11]
n_web_classes = len(web_classes)

fig, axes = plt.subplots(5, 3, figsize=(20,20))
for i in range(n_web_classes):
    image = mpimg.imread("German Traffic Signs/{0}.jpg".format(i + 1))
    web_images.append(cv2.resize(image,(32, 32), interpolation = cv2.INTER_AREA))
    axes[i,0].imshow(image)
    axes[i,0].set_title("Image {0}".format(i))
    axes[i,0].axis("off")
    axes[i,1].imshow(web_images[i])
    axes[i,1].set_title("Resized image {0}".format(i))
    axes[i,1].axis("off")
    prototype = X_train[class_index[web_classes[i]][100]].squeeze()
    axes[i,2].imshow(prototype)
    axes[i,2].set_title(labelmap[y_train[class_index[web_classes[i]][100]]])
    axes[i,2].axis("off")
plt.show()

# Finally, normalize the images:
web_images = np.array([normalize_image(i) for i in np.array(web_images).astype(np.float32)])
Left: web image, Center: web image scaled to 32x32, Right: example image from original data set in the same class

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.

Answer:

In [20]:
# Test the model against these new images (normalized data)
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    test_accuracy = evaluate(web_images, web_classes)
    print("Test Accuracy = {:.3f}".format(test_accuracy))
Test Accuracy = 0.600

Question 7

Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.

NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.

Answer:

It's able to perform relatively well, but the model doesn't seem to be tolerant of size and location changes for the detected objects. For example, the roundabout mandatory image is pretty close to the training set data, but the angle of the picture and other characteristics appear to have affected the outcome.

I blame this on limited input data, and in that I still don't understand how I can make my model more resilient to image deformation without necessarily feeding a huge amount of examples into the model. That's a question I'll throw in the Slack channel or the forums.

In [17]:
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.

zero_to_n = [i for i in range(n_classes)]

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    softmax = sess.run(tf.nn.softmax(logits), feed_dict={x: web_images, y: web_classes, keep_probability: 1.0})
    fig, axes = plt.subplots(5, 2, figsize=(20,20))
    for i in range(n_web_classes):
        axes[i,0].imshow(mpimg.imread("German Traffic Signs/{0}.jpg".format(i + 1)))
        axes[i,0].set_title(labelmap[web_classes[i]])
        axes[i,0].axis("off")
        axes[i,1].plot(zero_to_n, softmax[i])
        axes[i,1].set_title("Softmax for " + labelmap[y_train[class_index[web_classes[i]][100]]])
        axes[i,1].axis("on")
    plt.show()

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

Answer:

This is so cool! Visualizing the softmax and graphing examples of the top 3 hits for an image really makes this a very interesting tool for classification and for fine-tunning of the model.

In my case, the images that didn't get recognized were classified completely wrong, as if the model was super certain it was seeing something else. I'm thinking that my data augmentation is to blame for this and I could have done a better job. I could also have picked images that were even more similar to the ones in the training set, but then that wouldn't have been interesting would it?

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In [21]:
print("Top 3 hits")
print("Left: web image")

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    top_k = sess.run(tf.nn.top_k(tf.nn.softmax(logits), k=3),
                     feed_dict={x: web_images, y: web_classes, keep_probability: 1.0})

    fig, axes = plt.subplots(5, 4, figsize=(20,20))
    for i in range(n_web_classes):
        axes[i,0].imshow(mpimg.imread("German Traffic Signs/{0}.jpg".format(i + 1)))
        axes[i,0].set_title("Web image: " + labelmap[web_classes[i]])
        axes[i,0].axis("off")
        axes[i,1].imshow(X_train[class_index[top_k[1][i][0]]][100].squeeze())
        axes[i,1].set_title("1. {:} ({:.1f}%)".format(labelmap[top_k[1][i][0]],top_k[0][i][0]*100))
        axes[i,1].axis("off")
        axes[i,2].imshow(X_train[class_index[top_k[1][i][1]]][100].squeeze())
        axes[i,2].set_title("2. {:} ({:.1f}%)".format(labelmap[top_k[1][i][1]],top_k[0][i][1]*100))
        axes[i,2].axis("off")
        axes[i,3].imshow(X_train[class_index[top_k[1][i][2]]][100].squeeze())
        axes[i,3].set_title("3. {:} ({:.1f}%)".format(labelmap[top_k[1][i][2]],top_k[0][i][2]*100))
        axes[i,3].axis("off")
    plt.show()
Top 3 hits
Left: web image